A prospect list merged from several sources usually contains the same company more than once. One row comes from a directory with https://www.shreepack.co.in/contact, another from a trade portal as shreepack.co.in with a sales email, and a third with only a Gmail address and a phone number written as 098290-12345. If you send outreach from that list, the same owner gets three messages in a day, and that is the fastest way to look like a bot.
This post shows how to dedupe a company list in Python with a small, standard-library-only script that collapses those rows into one company. It keys each row on a root domain and a phone key, then merges rows that share any key using union-find, so A matches B by domain and B matches C by phone and all three end up together. I tested every block below with Python 3.14; it needs 3.9 or newer for str.removeprefix.
I work on Mantis, a prospecting tool for B2B teams. Duplicate rows are the first thing I clean in any merged list, and this is the approach I would hand to a developer starting from scratch.
How to dedupe a company list in Python: the plan
- Input: a CSV with
name, website, email, phone(any column can be empty) - Output: one row per company, plus a
merged_fromcount - Rules: two rows are the same company if they share a root domain or a phone key, unless that key is shared by too many rows
Save the blocks in order as dedupe.py.
Step 1: imports and the lists that matter
import csv
import io
import re
from collections import Counter
from urllib.parse import urlsplit
# A few multi-label suffixes from the Public Suffix List. In production,
# load the full list (publicsuffix.org/list/public_suffix_list.dat).
MULTI_SUFFIXES = {"co.in", "firm.in", "net.in", "org.in", "gen.in", "ind.in",
"co.uk", "com.au"}
# Hosts that never identify one company.
NOT_A_COMPANY = {"gmail.com", "yahoo.com", "yahoo.co.in", "outlook.com",
"hotmail.com", "rediffmail.com", "facebook.com",
"instagram.com", "linktr.ee", "wa.me"}
# Platforms where each subdomain is a different business.
TENANT_HOSTS = {"business.site", "wixsite.com", "blogspot.com"}
Three small sets carry most of the correctness:
-
MULTI_SUFFIXES: a public suffix is a name "under which Internet users can (or historically could) directly register names", and the examples given are com and co.uk. India has many of these: the Public Suffix List includes co.in, firm.in, gen.in, ind.in, net.in and org.in. I keep a handful inline to stay dependency-free; for real data, load the full list file. -
NOT_A_COMPANY: Gmail, Rediffmail, Facebook and Linktree never identify a single business. If you key ongmail.com, every sole trader in the file merges into one giant blob. -
TENANT_HOSTS: on hosts likebusiness.site, the subdomain is the business, so the full host is the key.
Step 2: root domain
def root_domain(value):
"""'https://WWW.Shop.Example.co.in/contact' -> 'example.co.in'."""
if not value:
return None
value = value.strip().lower()
if "@" in value and "/" not in value: # an email address
value = value.rsplit("@", 1)[1]
if "//" not in value: # urlsplit needs '//' to find the host
value = "//" + value
host = urlsplit(value).hostname
if not host or "." not in host:
return None
labels = host.split(".")
if ".".join(labels[-2:]) in TENANT_HOSTS:
return host.removeprefix("www.")
if ".".join(labels[-2:]) in MULTI_SUFFIXES:
root = ".".join(labels[-3:])
else:
root = ".".join(labels[-2:])
if root in NOT_A_COMPANY or host in NOT_A_COMPANY:
return None
return root
Two traps here.
First, urlsplit("shreepack.co.in").hostname returns None. The Python docs say urlsplit "recognizes a netloc only if it is properly introduced by '//'" and otherwise treats the input as a path. Prefixing // fixes bare domains without inventing a scheme.
Second, the naive "last two labels" rule. Run it on www.shreepack.co.in and ops.acme.co.in and both come back as co.in. On an Indian list that one bug merges every .co.in company into a single record. The suffix check takes three labels instead.
Step 3: a phone key (not a full phone cleaner)
def phone_key(value, country="91"):
"""'+91 98290 12345', '098290-12345', '9829012345' -> '+919829012345'."""
if not value:
return None
raw = value.strip()
digits = re.sub(r"\D", "", raw)
if raw.startswith("+") or digits.startswith("00"):
digits = digits.lstrip("0") # already international
else:
if digits.startswith(country) and len(digits) > 10:
digits = digits[len(country):]
digits = country + digits.lstrip("0") # drop trunk prefix 0
if len(digits) < 10:
return None
return "+" + digits
This is only a matching key, not a validator. It strips formatting, drops the trunk prefix 0 used for STD dialling, and prefixes India's country code 91 so +91 98290 12345, 098290-12345 and 9829012345 produce the same key. Short strings return None so they never match anything.
My first version had a bug that the tests caught: it turned the UK number +44 20 7946 0958 into +91442079460958, because it prefixed 91 to everything. Numbers that already start with + or 00 are now left as international. If your list is mostly one country, change the country default; if it is genuinely mixed, use a proper library such as phonenumbers for the key.
Step 4: merge with union-find
class UnionFind:
def __init__(self):
self.parent = {}
def find(self, x):
self.parent.setdefault(x, x)
while self.parent[x] != x:
self.parent[x] = self.parent[self.parent[x]]
x = self.parent[x]
return x
def union(self, a, b):
self.parent[self.find(a)] = self.find(b)
def row_keys(row):
keys = {root_domain(row.get("website")), root_domain(row.get("email")),
phone_key(row.get("phone"))}
return keys - {None}
def dedupe(rows, max_rows_per_key=5):
counts = Counter(k for row in rows for k in row_keys(row))
uf = UnionFind()
first_seen = {}
for i, row in enumerate(rows):
uf.find(i)
for key in row_keys(row):
if counts[key] > max_rows_per_key: # shared switchboard, agency domain...
continue
if key in first_seen:
uf.union(i, first_seen[key])
else:
first_seen[key] = i
groups = {}
for i in range(len(rows)):
groups.setdefault(uf.find(i), []).append(rows[i])
return list(groups.values())
Why union-find and not a groupby on domain? Because duplicates chain. Row 1 and row 2 share shreepack.co.in. Row 3 has only Gmail (ignored) and a phone number that matches row 1. A group-by on one column never joins row 3 to row 2; union-find does, because each shared key links two rows and the links are transitive.
The max_rows_per_key cap is the part people skip. Some keys can be shared by many unrelated businesses: an accountant's number printed on clients' listings, a shared call centre, or an agency domain hosting many client pages. Without a cap, one such key glues dozens of companies together. In my test, seven rows sharing one number stayed seven separate groups with the default cap and collapsed into a single group when I raised it. Print the keys that hit the cap and look at them; they are usually worth a manual check.
Step 5: merge each group and run it
def merge(group):
"""Keep the first non-empty value per column."""
out = {}
for row in group:
for col, val in row.items():
if val and not out.get(col):
out[col] = val
out["merged_from"] = len(group)
return out
SAMPLE = """name,website,email,phone
Shree Packaging,https://www.shreepack.co.in/contact,,+91 98290 12345
Shree Packaging Pvt Ltd,shreepack.co.in,sales@shreepack.co.in,
Shri Packaging (Kota),,shreepack@gmail.com,098290-12345
Kota Steel Traders,http://kotasteel.in,,0744 2390011
Kota Steel,,info@kotasteel.in,
Bharat Prints,https://www.facebook.com/bharatprints,,7742001122
Bharat Prints,,bharatprints@gmail.com,+91-77420-01122
Acme Logistics,acme.co.in,,
Acme Logistics,https://ops.acme.co.in,,
Nova Tiles,novatiles.business.site,,9001234567
"""
if __name__ == "__main__":
rows = list(csv.DictReader(io.StringIO(SAMPLE)))
groups = dedupe(rows)
print(f"{len(rows)} rows -> {len(groups)} companies")
for g in groups:
m = merge(g)
print(f"{m['merged_from']}x {m['name']:<20} {m.get('website','') or '-':<38} {m.get('phone','') or '-'}")
naive = lambda u: ".".join(urlsplit("//" + u).hostname.split(".")[-2:])
print("naive:", naive("www.shreepack.co.in"), naive("ops.acme.co.in"))
print(root_domain("novatiles.business.site"), root_domain("shop.example.co.uk"), root_domain("example.co.in"), root_domain("x.y.example.in"))
print(phone_key("0744 2390011"), phone_key("+91 98290 12345"), phone_key("12345"))
Output:
10 rows -> 5 companies
3x Shree Packaging https://www.shreepack.co.in/contact +91 98290 12345
2x Kota Steel Traders http://kotasteel.in 0744 2390011
2x Bharat Prints https://www.facebook.com/bharatprints 7742001122
2x Acme Logistics acme.co.in -
1x Nova Tiles novatiles.business.site 9001234567
naive: co.in co.in
novatiles.business.site example.co.uk example.co.in example.in
+917442390011 +919829012345 None
The sample rows are made up, but each mirrors a pattern that shows up in merged exports. Ten rows became five companies. Shree Packaging merged through domain and phone, Bharat Prints merged through phone alone because both its Facebook URL and its Gmail were ignored, and the two Acme rows merged because ops.acme.co.in and acme.co.in share a root.
What this does not do
- Fuzzy names. "Shree Packaging" and "Shri Packaging" are not compared at all; they merge here only because they share keys. Add name matching only as a tie-breaker inside a city, never as a key by itself, or two different "Sharma Traders" will merge.
- Chains and branches. Ten branches of one brand share a domain and will merge into one row. If you sell to branches, add the city or address to the domain key.
-
Picking the best value.
merge()keeps the first non-empty value. In production, prefer the most recently verified source.
FAQ
Why not just use pandas drop_duplicates? It only removes exact matches on the columns you name. The rows above are never exact matches, and it cannot chain matches across columns.
Should email domains count as company keys? Yes, once free-mail providers are excluded. A sales address at the company domain can be the only link between a directory row and a portal row.
Is a phone match enough on its own? Usually, for small businesses, but that is exactly why the shared-key cap exists. Review anything that hits it.
If you would rather start from a list that is already deduped, I wrote up how we build lead lists without buying databases here: how to get leads without buying lists.
Top comments (0)