Merge and deduplicate scraped datasets across runs (JS + Python)

I wrote this article with AI assistance. I ran every snippet on 5 Nov 2026 with Node 24 and Python 3.13 (and checked them on Node 18 and Python 3.9), and all output is copied from those runs. The sample data is fictional.

If you scrape the same source more than once, you get duplicates. A second run overlaps the first. Two search queries return the same company. A directory and a map listing describe one business in two formats. Before you load the data anywhere, you need one row per real-world thing.

Deduplication (dedup) means finding the rows that describe the same thing and keeping one. Merging means combining those rows into one row that has the best value for each field.

Who this is for: developers who collect data from scrapers, APIs or CSV exports and need to combine several runs or sources.

What you need: Node 18 or later for the JavaScript script, or Python 3.9 or later for the Python scripts. None of them needs any packages.

The sample data

Save this as leads.json. It holds eight company records from two scraper runs, on 1 and 4 November. The .example domains are reserved for documentation, so none of these companies is real.

[
  {"name": "Acme Sdn Bhd", "website": "https://www.acme.example/", "email": "[e-mail]", "phone": "[phone]", "city": "", "scrapedAt": "2026-11-01"},
  {"name": "Kopi Corner", "website": "http://kopicorner.example/menu?utm_source=maps", "email": "", "phone": "012-345 6789", "city": "Penang", "scrapedAt": "2026-11-01"},
  {"name": "Batik House", "website": "", "email": "[e-mail]", "phone": "", "city": "Melaka", "scrapedAt": "2026-11-01"},
  {"name": "ACME Sdn. Bhd.", "website": "acme.example", "email": "[e-mail] ", "phone": "03-1234 5678", "city": "Kuala Lumpur", "scrapedAt": "2026-11-04"},
  {"name": "Kopi Corner ", "website": "https://kopicorner.example/menu", "email": "[e-mail]", "phone": "[phone]", "city": "George Town", "scrapedAt": "2026-11-04"},
  {"name": "Batik House", "website": "https://batikhouse.example", "email": "", "phone": "06-222 3333", "city": "Melaka", "scrapedAt": "2026-11-04"},
  {"name": "Nasi Lemak Co", "website": "https://nasilemak.example", "email": "[e-mail]", "phone": "", "city": "Ipoh", "scrapedAt": "2026-11-04"},
  {"name": "Roti Studio", "website": "", "email": "[e-mail]", "phone": "", "city": "Ipoh", "scrapedAt": "2026-11-04"}
]

A person can see five companies here. Code can't yet. The two Acme rows differ in name, website, email and phone format. The two Batik House rows share no field except the name and city.

1. Choose a dedup key

A dedup key is the value that identifies a record. Two rows with the same key are duplicates, so choose the key first.

Good keys are stable and unique: an ID from the source (a place ID or a job ID), a product URL, or a company's website domain. Bad keys change between runs or are shared by unrelated records: names, page titles, timestamps, and the position of a row in the results.

For company data, the website domain is a good key. When a row has no website, the domain of a company email address is a fallback. Free email domains such as gmail.com are not a fallback, because thousands of unrelated companies share them. A row with no reliable key should stay as it is. Guessing would merge companies that only look alike.

2. Normalise the key, then deduplicate (JavaScript)

Normalising a value means rewriting it into one standard form, so that different spellings of the same thing compare as equal. dedupe.mjs applies these rules:

Value	Rule	Example
Any text	Trim and collapse whitespace	"Kopi Corner " → "Kopi Corner"
URL	Drop http(s):// and www., lowercase the host, drop the trailing slash and tracking parameters (utm_*, fbclid, gclid), sort the remaining parameters	http://www.Kopicorner.example/menu/?utm_source=maps&b=2&a=1 → kopicorner.example/menu?a=1&b=2
Email	Trim and lowercase	[e-mail] → [e-mail]
Phone	Keep the digits, compare the last 9	[phone] and 012-345 6789 → 123456789

The phone rule ignores country codes and the leading trunk 0. That suits data from one country. If your data mixes countries, two different numbers can share their last 9 digits, so parse them with a library such as libphonenumber instead.

The script below builds a key for each row and removes duplicates in one of two ways:

• Keep first keeps the first copy it sees. It is the simplest, and works in one pass over a stream.
• Keep newest keeps the copy with the latest scrapedAt date, so fresh data replaces old data.

Save it as dedupe.mjs:

// dedupe.mjs - remove duplicate records from several scraper runs.
// Node 18+. No dependencies. Usage: node dedupe.mjs leads.json
import { readFileSync } from 'node:fs';

const FREE_MAIL = new Set(['gmail.com', 'yahoo.com', 'hotmail.com', 'outlook.com']);
const TRACKING = /^(utm_|fbclid$|gclid$)/i;

const clean = (s) => String(s ?? '').trim().replace(/\s+/g, ' ');

function normUrl(s) {
  s = clean(s);
  if (!s) return '';
  const u = new URL(/^https?:\/\//i.test(s) ? s : `https://${s}`);
  const host = u.hostname.toLowerCase().replace(/^www\./, '');
  const params = [...u.searchParams].filter(([k]) => !TRACKING.test(k)).sort();
  const qs = params.length ? `?${new URLSearchParams(params)}` : '';
  return host + u.pathname.replace(/\/+$/, '') + qs;
}

const normEmail = (s) => clean(s).toLowerCase();
// Digits only, then the last 9 digits, so "[phone]" equals "012-345 6789".
const normPhone = (s) => clean(s).replace(/\D/g, '').slice(-9);

// The key: the website's domain, or the email's domain if it isn't free mail.
function keyOf(r) {
  const site = normUrl(r.website);
  if (site) return site.split('/')[0];
  const domain = normEmail(r.email).split('@')[1];
  if (domain && !FREE_MAIL.has(domain)) return domain;
  return null; // no reliable key: keep the row as it is
}

function dedupe(rows, keep) {
  const byKey = new Map();
  const out = [];
  for (const r of rows) {
    const k = keyOf(r);
    if (k === null) { out.push(r); continue; }
    const prev = byKey.get(k);
    if (!prev) { byKey.set(k, r); out.push(r); continue; }
    if (keep === 'newest' && r.scrapedAt > prev.scrapedAt) {
      out[out.indexOf(prev)] = r;
      byKey.set(k, r);
    }
  }
  return out;
}

const rows = JSON.parse(readFileSync(process.argv[2], 'utf8'));
console.log('normUrl:', normUrl('http://www.Kopicorner.example/menu/?utm_source=maps&b=2&a=1'));
console.log('normPhone:', normPhone('[phone]'), normPhone('012-345 6789'));
console.log('keys:', rows.map((r) => keyOf(r) ?? '(none)').join(', '));
for (const keep of ['first', 'newest']) {
  const out = dedupe(rows, keep);
  console.log(`\nkeep=${keep}: ${rows.length} rows -> ${out.length}`);
  for (const r of out) console.log(`  ${r.scrapedAt}  ${clean(r.name).padEnd(15)} ${r.city || '-'}`);
}

Run it:

node dedupe.mjs leads.json

Output:

normUrl: kopicorner.example/menu?a=1&b=2
normPhone: 123456789 123456789
keys: acme.example, kopicorner.example, batikhouse.example, acme.example, kopicorner.example, batikhouse.example, nasilemak.example, (none)

keep=first: 8 rows -> 5
  2026-11-01  Acme Sdn Bhd    -
  2026-11-01  Kopi Corner     Penang
  2026-11-01  Batik House     Melaka
  2026-11-04  Nasi Lemak Co   Ipoh
  2026-11-04  Roti Studio     Ipoh

keep=newest: 8 rows -> 5
  2026-11-04  ACME Sdn. Bhd.  Kuala Lumpur
  2026-11-04  Kopi Corner     George Town
  2026-11-04  Batik House     Melaka
  2026-11-04  Nasi Lemak Co   Ipoh
  2026-11-04  Roti Studio     Ipoh

The key turned eight rows into five. The first Batik House row has no website, so the script used its email domain, and both Batik House rows got the key batikhouse.example. Roti Studio has only a Gmail address, so it got no key and stayed as a separate row.

Both modes still lose data. "Keep first" lost Kuala Lumpur, the city in Acme's second row. "Keep newest" lost the email address in Batik House's first row. To keep both, merge the duplicates.

3. Merge duplicates into one complete row (Python)

The script below groups rows by the same key, then builds one row per group:

1. It sorts each group newest first.
2. It takes each field from the newest copy that has a value, so older copies only fill gaps.
3. It records a conflict when two copies have different non-empty values for a field, so you can review them.

Two values only conflict if they still differ after normalisation. Without that step, [phone] and 012-345 6789 would conflict. The script uses the phone rule for phone, and case and whitespace rules for most fields. For website it uses a simpler URL rule than dedupe.mjs: it drops the scheme, www., the whole query string and the trailing slash.

Save it as merge.py:

# merge.py - merge duplicate records into one complete row per company.
# Python 3.9+. Standard library only. Usage: python3 merge.py leads.json
import json, re, sys

FREE_MAIL = {"gmail.com", "yahoo.com", "hotmail.com", "outlook.com"}

def clean(s):
    return re.sub(r"\s+", " ", str(s or "")).strip()

def key_of(r):
    site = clean(r.get("website")).lower()
    if site:
        site = re.sub(r"^https?://", "", site)
        return re.sub(r"^www\.", "", site).split("/")[0].split("?")[0]
    domain = clean(r.get("email")).lower().partition("@")[2]
    return domain if domain and domain not in FREE_MAIL else None

def norm_url(s):
    s = re.sub(r"^https?://(www\.)?", "", clean(s).lower())
    return s.split("?")[0].rstrip("/")

NORMALISE = {
    "website": norm_url,
    "phone": lambda s: re.sub(r"\D", "", str(s))[-9:],
}

def same(field, a, b):
    # Compare with the field's own normaliser; default: ignore case and spaces.
    f = NORMALISE.get(field, lambda s: clean(s).lower())
    return f(a) == f(b)

def merge(group):
    # Newest copy first, so its non-empty values win. Older copies only fill gaps.
    group = sorted(group, key=lambda r: r["scrapedAt"], reverse=True)
    out, conflicts = {}, set()
    for r in group:
        for field, value in r.items():
            if not clean(value):
                out.setdefault(field, value)
            elif not clean(out.get(field)):
                out[field] = clean(value)
            elif field != "scrapedAt" and not same(field, out[field], value):
                conflicts.add(field)
    out["_copies"] = len(group)
    out["_conflicts"] = sorted(conflicts)
    return out

rows = json.load(open(sys.argv[1]))
groups, singles = {}, []
for r in rows:
    k = key_of(r)
    if k is None:
        singles.append(r)
    else:
        groups.setdefault(k, []).append(r)

merged = [merge(g) for g in groups.values()] + singles
print(f"{len(rows)} rows -> {len(merged)}")
for m in merged:
    print(json.dumps(m, ensure_ascii=False))

Run it:

python3 merge.py leads.json

Output:

8 rows -> 5
{"name": "ACME Sdn. Bhd.", "website": "acme.example", "email": "[e-mail]", "phone": "03-1234 5678", "city": "Kuala Lumpur", "scrapedAt": "2026-11-04", "_copies": 2, "_conflicts": ["name"]}
{"name": "Kopi Corner", "website": "https://kopicorner.example/menu", "email": "[e-mail]", "phone": "[phone]", "city": "George Town", "scrapedAt": "2026-11-04", "_copies": 2, "_conflicts": ["city"]}
{"name": "Batik House", "website": "https://batikhouse.example", "email": "[e-mail]", "phone": "06-222 3333", "city": "Melaka", "scrapedAt": "2026-11-04", "_copies": 2, "_conflicts": []}
{"name": "Nasi Lemak Co", "website": "https://nasilemak.example", "email": "[e-mail]", "phone": "", "city": "Ipoh", "scrapedAt": "2026-11-04", "_copies": 1, "_conflicts": []}
{"name": "Roti Studio", "website": "", "email": "[e-mail]", "phone": "", "city": "Ipoh", "scrapedAt": "2026-11-04"}

Every company now has one row. No field is empty if any copy had a value, and values that disagree are flagged:

• Acme takes every value from its newest copy. Its only conflict is name, because ACME Sdn. Bhd. and Acme Sdn Bhd differ by two dots.
• Kopi Corner has a real conflict: the city was Penang on 1 November and George Town on 4 November. George Town is the capital of Penang state, so both may be correct. Only a person or a better source can decide.
• Batik House combines the website and phone from 4 November with the email from 1 November.

_copies and _conflicts let you check the result. Sort by _conflicts and review those rows before you trust the merge.

4. Don't use fuzzy matching as your key

Fuzzy matching treats two values as the same when they are similar, not equal. It is tempting for names. Here is Python's built-in difflib scoring four pairs of names. A score of 1.0 means identical.

Save it as fuzzy.py:

# fuzzy.py - why similar names make a risky dedup key.
from difflib import SequenceMatcher

pairs = [
    ("Acme Sdn Bhd", "ACME Sdn. Bhd."),          # same company
    ("Kopi Corner", "Kopi Korner"),               # a typo, or a different shop?
    ("Kopi Corner Penang", "Kopi Corner Ipoh"),   # two branches
    ("Nasi Lemak Co", "Roti Studio"),             # clearly different
]
for a, b in pairs:
    score = SequenceMatcher(None, a.lower(), b.lower()).ratio()
    print(f"{score:.2f}  {a!r} vs {b!r}")

Output:

0.92  'Acme Sdn Bhd' vs 'ACME Sdn. Bhd.'
0.91  'Kopi Corner' vs 'Kopi Korner'
0.76  'Kopi Corner Penang' vs 'Kopi Corner Ipoh'
0.25  'Nasi Lemak Co' vs 'Roti Studio'

The same company scored 0.92. A typo, or possibly a different shop, scored 0.91. Only 0.01 separates them, so a threshold that merges Acme but not Kopi Korner works for these names by luck and fails on the next pair. Once two companies are merged, the data that separated them is gone.

Use fuzzy scores to suggest duplicates for a person to review. Use exact matching on a normalised key to merge them.

5. Checklist for your own data

• Pick the key before you write code. Prefer IDs and URLs from the source. Avoid names.
• Normalise per field type. URLs, emails and phones each need their own rule.
• Don't guess keys. Rows with no reliable key, such as free email domains, stay separate.
• Decide which copy wins. Newest-first suits data that changes. First-seen suits data that doesn't.
• Merge instead of dropping when copies hold different fields.
• Flag conflicts instead of resolving them silently.
• Keep the raw runs so you can rebuild the merged file when a rule changes.

About the hosted tool (disclosure: this is my own project)

I built and sell a hosted version of this workflow: Dataset Dedup, Merge & Compare on Apify, a platform for running scrapers and automation tools. It reads one or more Apify datasets in the order you list them and deduplicates by one or more fields, which can be nested (location.city). It can keep the first copy, the last copy, or merge all copies. Merged rows record _mergedCount and _mergedFrom, plus _filledFrom and _conflictingFields when they apply. Optional matching rules cover case, whitespace, accents, URLs, phone numbers (digits only, last 9, as in section 2) and domains (https://www.acme.com/contact, [e-mail] and acme.com all match). It uses normalised matching, not fuzzy matching, for the reasons in section 4.

Two features go beyond the scripts above. It can remember the keys it has output under a memory name, so later runs return only new items. It can also compare two datasets or CSV/JSON files and report field-level differences.

It differs from merge.py in two ways:

• On a conflict, it keeps the first non-empty value in dataset order. To make the newest values win, list the newest dataset first.
• A dedup key uses all the fields you list together. It has no fallback from website to email. Rows with an empty key are kept as separate rows. To get a fallback, add a key column to your data first.

I ran the tool's dedup engine locally (not the hosted actor) on the sample data, with the newest run listed first, these options, and the default case and whitespace rules:

{ "dedupFields": ["website"], "keep": "merge", "normalizeDomains": true, "normalizePhones": true }

Three lines of the output:

8 rows -> 6 (merged groups 2, empty-key rows 2)
{"name":"ACME Sdn. Bhd.","website":"acme.example","email":"[e-mail] ","phone":"03-1234 5678","city":"Kuala Lumpur","scrapedAt":"2026-11-04","_mergedCount":2,"_mergedFrom":[{"datasetId":"run2","index":0},{"datasetId":"run1","index":0}],"_conflictingFields":["name","scrapedAt"]}
{"name":"Batik House","website":"","email":"[e-mail]","phone":"","city":"Melaka","scrapedAt":"2026-11-01"}

It merged Acme and Kopi Corner and flagged the same conflicts as merge.py, plus scrapedAt. The first Batik House row has no website, so it stayed separate. It also keeps values as they were scraped ("Kopi Corner " keeps its trailing space), because normalisation only applies to matching.

Pricing is $0.20 per 1,000 output rows plus $0.001 per run. Removed duplicates, filtered rows, already-seen rows and error records are free. With no datasets selected, it runs a free demo on a sample dataset.

You don't need the hosted version: the scripts in this post work on their own.

Summary

To deduplicate data from several runs, choose a stable key, normalise it per field type, and keep rows without a reliable key separate. Merge duplicates newest first so no field is lost, and flag conflicts for review. Use fuzzy matching to suggest duplicates, not to merge them.

What do you use as a dedup key for messy data? If you have a normalisation rule that saved you from a bad merge, please share it in the comments.