How to Build a B2B Contact List from Company Websites Responsibly

You have a list of target companies from a trade-show catalogue or a CRM export, and you need each one's general business contact: the sales inbox, the switchboard, the official LINE account. The manual way is to open every homepage, find "Contact" or "關於我們", copy what's there into a spreadsheet, and repeat. It's slow, and you end up with phone numbers in five formats and no record of where anything came from.

This post shows how to automate that responsibly: follow robots.txt, read only a few pages per site, collect only public business contact info, and keep the source of every value. A runnable Python example is included.

1. Decide what you are collecting (and what you are not)

The target is company contact points: role-based addresses (info@, sales@), main lines, fax and toll-free numbers, and the company's own Facebook page and LINE official account. A named employee's email or a sole trader's mobile is personal data under GDPR and Taiwan's PDPA, even when it's on a public page.

• Minimise. Keep only the columns you'll use.
• Keep provenance. Store the URL each value came from, so you can answer "where did you get my address?" and re-check stale data.

2. Crawl politely: robots.txt and Crawl-delay

Before you fetch anything, read /robots.txt and follow it (RFC 9309):

• Skip every path that is Disallowed for your user agent.
• Honour Crawl-delay if it's set. urllib.robotparser exposes it as crawl_delay(). Without it, still wait at least a second per host.
• If robots.txt returns a 5xx error, don't crawl that site. RFC 9309 says to treat a server error as "complete disallow". A 404 means there are no rules.
• Send an honest User-Agent that says who you are. Don't log in, don't rotate proxies to look like someone else, and stop if you hit a CAPTCHA or bot challenge. A site that blocks you has answered your question.

3. Find the contact and about pages

Contact details usually sit in the homepage header or footer, a contact page and an about page, so you don't need a full crawl. Fetch the homepage, pick same-site links whose URL or anchor text matches contact, about, 聯絡, 關於 or お問い合わせ, and visit a few. About four pages per domain covers most small company sites. If nothing matches, try /contact and /about directly.

4. Extract emails, phones, LINE and social links

Emails. Take mailto: links first, then scan visible text, and filter obvious junk: asset names ([e-mail]) and placeholders ([e-mail]). If a site deliberately obfuscates its address, respect that and don't decode it.

Phones. Regexes alone will match dates, order numbers and prices. Two rules help a lot:

1. Require context. Only parse a phone number from a tel: link or a line that has a label nearby: Tel, Phone, 電話, 客服, 傳真, Fax.
2. Let libphonenumber do the parsing. The phonenumbers package (a Python port of Google's libphonenumber) understands Taiwanese formats such as [phone], 02-2345-6789, [phone] and 0800-000-123. It normalises all of them to E.164 ([phone]), which is what makes de-duplication work.

In Taiwan, capture extensions (分機 123, 轉 123) separately, and strip the 8-digit tax ID (統一編號) that often sits next to the number in the footer.

LINE. Businesses in Taiwan and Japan often publish a LINE official account instead of an email. Look for line.me/R/ti/p/@account, lin.ee short links and page.line.me.

Social profiles. Keep the company's own profiles and drop share buttons (/intent/tweet, /sharer).

5. De-duplicate with provenance

The same number often appears on three pages in three formats. Key phones by E.164 (plus extension) and emails by their lower-cased form, and keep a list of source URLs per key: one record per domain, with an audit trail.

6. A runnable example

The script below puts sections 2–5 together using only the standard library plus phonenumbers. To test it safely, I ran it against a fictional company site served from my own machine. All addresses use example.com, and the numbers are made up. Don't point a test script at real third-party sites.

The test site has a robots.txt that blocks /private/ and sets a crawl delay:

User-agent: *
Disallow: /private/
Crawl-delay: 1

The homepage links to an about page, a contact page and a page under /private/. Its footer puts a phone number and a tax ID on the same line:

<!doctype html><html lang="zh-Hant"><head><meta charset="utf-8"><title>Example Trading Co.</title></head>
<body>
<header><a href="/">Home</a> <a href="/products.html">Products</a> <a href="/about-us.html">關於我們</a> <a href="/contact.html">聯絡我們</a></header>
<p>Fictional demo company.</p>
<img src="/img/[e-mail]" alt="logo">
<a href="https://twitter.com/intent/tweet?url=https://example.com">Share on X</a>
<footer>
  <a href="https://www.facebook.com/exampletrading">Facebook</a>
  <a href="https://lin.ee/AbCdEfG">加入 LINE 好友</a>
  <a href="/private/contact-staff.html">Staff only</a>
  電話 [phone]｜統一編號 12345678
</footer>
</body></html>

The contact page mixes contacts with things that must not be extracted (an order number, a date, a placeholder email, an image name):

<!doctype html><html lang="zh-Hant"><head><meta charset="utf-8"><title>聯絡我們</title></head>
<body>
<h1>聯絡我們</h1>
<p>業務洽詢：<a href="mailto:[e-mail]">[e-mail]</a></p>
<p>客服信箱：[e-mail]</p>
<p>電話：<a href="tel:[phone]">[phone]</a> 分機 123</p>
<p>傳真：[phone]</p>
<p>免付費客服：0800-000-123</p>
<p>Order no. 2026101700123, updated 2026-10-17</p>
<p>Example form text: [e-mail] (image: [e-mail])</p>
<p>LINE 官方帳號：<a href="https://line.me/R/ti/p/@exampletrading">@exampletrading</a></p>
<footer><a href="https://www.facebook.com/exampletrading">Facebook</a></footer>
</body></html>

The extractor, extract_contacts.py (pip install phonenumbers):

"""Polite public-business-contact extractor (tutorial demo).

Reads a site's robots.txt, honours Disallow and Crawl-delay, visits the homepage
plus up to 3 contact/about pages, and returns one de-duplicated record per domain.
Only run it on sites you are allowed to crawl. Demo data is fictional.
"""
import json, re, sys, time, urllib.error, urllib.request, urllib.robotparser
from html.parser import HTMLParser
from urllib.parse import urljoin, urlsplit
import phonenumbers

UA = "ContactListTutorialBot/0.1 (+https://example.com/bot)"
MAX_PAGES = 4
PAGE_HINTS = re.compile(r"contact|about|聯絡|聯繫|關於|お問い合わせ", re.I)
EMAIL = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,24}")
JUNK_EMAIL = re.compile(r"\.(png|jpe?g|gif|svg|webp)$|^(you|name|user|email)@(example|domain)\.", re.I)
PHONE_CONTEXT = re.compile(r"tel|phone|電話|客服|傳真|fax|免付費", re.I)
FAX = re.compile(r"傳真|fax", re.I)
EXT = re.compile(r"^\s*(?:分機|轉|ext\.?|#)\s*(\d{1,6})", re.I)
LINE = re.compile(r"https?://(?:lin\.ee|line\.me|page\.line\.me)/[^\s\"'<>]+", re.I)
TAX_ID = re.compile(r"統一編號\s*[:：]?\s*\d{8}")
SHARE = re.compile(r"/(?:intent|share|sharer)", re.I)
SOCIAL = re.compile(r"https?://(?:www\.)?(facebook|instagram|linkedin|x|twitter|youtube)\.com/[^\s\"'<>]+", re.I)


class Page(HTMLParser):
    """Collects visible text (one line per block) and link targets."""
    def __init__(self):
        super().__init__(); self.links, self.lines, self._buf = [], [], []
    def handle_starttag(self, tag, attrs):
        href = dict(attrs).get("href")
        if href: self.links.append(href)
        if tag in ("p", "br", "li", "div", "footer", "header", "h1", "h2", "tr"): self._flush()
    def handle_data(self, data): self._buf.append(data)
    def _flush(self):
        line = " ".join("".join(self._buf).split())
        if line: self.lines.append(line)
        self._buf = []


def fetch(url):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    with urllib.request.urlopen(req, timeout=10) as r:
        return r.status, r.read().decode("utf-8", "replace")


def load_robots(base):
    rp = urllib.robotparser.RobotFileParser()
    try:
        _, body = fetch(base + "/robots.txt")
        rp.parse(body.splitlines())
    except urllib.error.HTTPError as e:
        if e.code >= 500:
            return None          # server error: treat as "do not crawl"
        rp.parse([])             # 404 etc.: no rules, everything allowed
    return rp


def extract(start_url, region="TW"):
    parts = urlsplit(start_url); base = f"{parts.scheme}://{parts.netloc}"
    record = {"domain": parts.hostname, "status": "ok", "pages": [], "skippedByRobots": [],
              "emails": {}, "phones": {}, "line": {}, "socials": {}, "junkFiltered": 0}
    rp = load_robots(base)
    if rp is None:
        record["status"] = "robots-unavailable"; return record
    delay = rp.crawl_delay(UA) or 1.0           # at least 1 s between requests
    queue, seen = [base + "/"], set()
    while queue and len(record["pages"]) < MAX_PAGES:
        url = queue.pop(0)
        if url in seen: continue
        seen.add(url)
        if not rp.can_fetch(UA, url):
            record["skippedByRobots"].append(url); continue
        if record["pages"]: time.sleep(delay)
        try:
            status, html = fetch(url)
        except urllib.error.URLError as e:
            record["pages"].append({"url": url, "error": str(e)}); continue
        record["pages"].append({"url": url, "http": status})
        page = Page(); page.feed(html); page._flush()
        collect(page, url, region, record)
        if url == base + "/":                    # discover contact/about pages from the homepage
            for href in page.links:
                full = urljoin(url, href).split("#")[0]
                if urlsplit(full).netloc == parts.netloc and PAGE_HINTS.search(href):
                    queue.append(full)
    if not (record["emails"] or record["phones"] or record["line"] or record["socials"]):
        record["status"] = "no-contacts-found"
    for key in ("emails", "phones", "line", "socials"):    # sets -> sorted lists for JSON
        record[key] = {k: sorted(v) if isinstance(v, set) else v for k, v in record[key].items()}
    return record


def add(bucket, key, source):
    bucket.setdefault(key, set()).add(source)


def collect(page, url, region, rec):
    for href in page.links:
        if href.startswith("mailto:"):
            add(rec["emails"], href[7:].split("?")[0].lower(), url)
        elif LINE.match(href):
            add(rec["line"], href, url)
        elif SOCIAL.match(href) and not SHARE.search(href):
            add(rec["socials"], href.rstrip("/"), url)
    for line in page.lines:
        for e in EMAIL.findall(line):
            if not JUNK_EMAIL.search(e):
                add(rec["emails"], e.lower(), url)
            else:
                rec["junkFiltered"] += 1
        line = TAX_ID.sub(" ", line)                # 統一編號 (8-digit tax ID) is not a phone
        if not PHONE_CONTEXT.search(line):
            continue                                # phones need context (Tel, 電話, 傳真 ...)
        for m in phonenumbers.PhoneNumberMatcher(line, region):
            e164 = phonenumbers.format_number(m.number, phonenumbers.PhoneNumberFormat.E164)
            ext = EXT.match(line[m.end:])
            key = e164 + (f";ext={ext.group(1)}" if ext else "")
            entry = rec["phones"].setdefault(key, {"kind": "fax" if FAX.search(line) else "phone", "sources": []})
            if url not in entry["sources"]: entry["sources"].append(url)


if __name__ == "__main__":
    print(json.dumps(extract(sys.argv[1]), ensure_ascii=False, indent=2))

Serve the site and run it:

python3 -m http.server 8000 --bind 127.0.0.1 --directory site &
python3 extract_contacts.py http://127.0.0.1:8000/

Real output (the demo script run_demo.sh picks a free port and appends the server log):

{
  "domain": "127.0.0.1",
  "status": "ok",
  "pages": [
    {
      "url": "http://127.0.0.1:48023/",
      "http": 200
    },
    {
      "url": "http://127.0.0.1:48023/about-us.html",
      "http": 200
    },
    {
      "url": "http://127.0.0.1:48023/contact.html",
      "http": 200
    }
  ],
  "skippedByRobots": [
    "http://127.0.0.1:48023/private/contact-staff.html"
  ],
  "emails": {
    "[e-mail]": [
      "http://127.0.0.1:48023/about-us.html",
      "http://127.0.0.1:48023/contact.html"
    ],
    "[e-mail]": [
      "http://127.0.0.1:48023/contact.html"
    ]
  },
  "phones": {
    "[phone]": {
      "kind": "phone",
      "sources": [
        "http://127.0.0.1:48023/",
        "http://127.0.0.1:48023/about-us.html"
      ]
    },
    "[phone];ext=123": {
      "kind": "phone",
      "sources": [
        "http://127.0.0.1:48023/contact.html"
      ]
    },
    "[phone]": {
      "kind": "fax",
      "sources": [
        "http://127.0.0.1:48023/contact.html"
      ]
    },
    "[phone]": {
      "kind": "phone",
      "sources": [
        "http://127.0.0.1:48023/contact.html"
      ]
    }
  },
  "line": {
    "https://lin.ee/AbCdEfG": [
      "http://127.0.0.1:48023/"
    ],
    "https://line.me/R/ti/p/@exampletrading": [
      "http://127.0.0.1:48023/contact.html"
    ]
  },
  "socials": {
    "https://www.facebook.com/exampletrading": [
      "http://127.0.0.1:48023/",
      "http://127.0.0.1:48023/contact.html"
    ]
  },
  "junkFiltered": 2
}
--- server access log (shows robots.txt first and the Crawl-delay spacing) ---
[17/Oct/2026 16:31:02] "GET /robots.txt HTTP/1.1" 200 -
[17/Oct/2026 16:31:02] "GET / HTTP/1.1" 200 -
[17/Oct/2026 16:31:03] "GET /about-us.html HTTP/1.1" 200 -
[17/Oct/2026 16:31:04] "GET /contact.html HTTP/1.1" 200 -

What to notice:

• robots.txt was fetched first, and /private/contact-staff.html was skipped even though it matched the "contact" hint.
• Requests are about one second apart, as Crawl-delay: 1 asks.
• [phone] and [phone] became one E.164 key with two sources. The 分機 123 version kept its own key, and the fax line was labelled fax.
• The tax ID, order number, date, placeholder email and image file name produced no contacts (junkFiltered: 2 counts the two email-like strings).
• The share-on-X link was dropped, and both LINE link styles (lin.ee and line.me/R/ti/p/@…) were kept.

It's a teaching script. It reads phones from labelled text (not tel: links), doesn't try /contact when nothing matches, and has no JavaScript rendering, retries or parallelism.

7. GDPR and PDPA: your responsibilities

This section is general information, not legal advice.

• You decide which sites to read and how to use the results, so compliance is your responsibility, not the tool's.
• GDPR (EU/EEA and UK). You need a lawful basis, usually legitimate interest (Art. 6(1)(f)) backed by a documented balancing test. If any personal data ends up in the list, you have to tell those people within a reasonable time (Art. 14) and honour objections (Art. 21). National ePrivacy rules often require prior consent before you send marketing email to individuals.
• Taiwan PDPA (個人資料保護法). Data the person made public themselves, or that was lawfully made public, may be collected, but only for a specific purpose (Art. 19). The notice obligation can be waived for such data (Art. 9). If you use it for marketing, you must offer a way to refuse at first contact and stop as soon as the person objects (Art. 20).
• In practice: prefer role-based addresses, keep source URLs, honour "no solicitation" notices, offer an opt-out in every first message, delete what you don't need, and never build profiles of individuals.

Checklist

• [ ] Company contact points only
• [ ] Fetch robots.txt first; skip disallowed paths; 5xx means don't crawl
• [ ] Crawl-delay (≥1 s per host), honest User-Agent, no logins or proxies, stop at challenges
• [ ] Homepage plus 3 contact/about pages per domain
• [ ] Phones only with context; normalise to E.164; keep extensions; strip 統一編號
• [ ] Filter junk emails; don't decode obfuscated ones
• [ ] Keep LINE official account links; drop share buttons
• [ ] De-duplicate by normalised key and keep source URLs
• [ ] Document lawful basis and purpose; offer an opt-out

This article was written with AI assistance. The code was run and the output above is copied from that run.