I generated 500 messy scraped leads and tried to clean them: what broke

Disclosure: I built the Actor mentioned at the end of this post. The test data below is synthetic, and I say so wherever it matters.

TL;DR: on a synthetic 500-row file, the cleaner returned the 400 real businesses, merged the 90 planted duplicates and made no false merges. Getting there took three dedupe fixes (shared inboxes, shared name prefixes, numbered branches) and two deployment fixes (dataset permissions, monorepo bundling).

Scraped lead lists have a reputation for being messy, and everyone nods at that. I wanted numbers, so I built a fixture and tried to clean it with the tool I am writing: an Apify Actor that normalizes, dedupes and scores business records.

This post is about what broke. The headline results are at the end of the first section, but the bugs are the useful part.

The fixture

I did not want to scrape a real site just to test a cleaner, and I did not want to publish someone else's business data. So I wrote a generator. It is deterministic (seeded), and it produces:

• 400 distinct businesses with unique names, written out as 500 rows
• 3 different scraper field schemas mixed in the same file: title / emailAddress / phone / website; Company Name / E-mail / Phone Number / URL; and name / emails[] / phones[] / site (with street_address and town)
• 90 intentional duplicates: a second sighting of an existing business, pushed through a random schema, with a typo in the name, a missing email, or a missing phone and website
• noise on top: ALL-CAPS and doubled spaces, tracking parameters, doubled @@, mailto: prefixes, mojibake accents, disposable and mistyped email domains
• 10 junk rows with almost nothing usable in them

Here is a real duplicate pair from the file (same business, two schemas, one invalid email, one missing website):

{ "Company Name": "  Pioneer  Plumbing ", "E-mail": "anna.k@@pioneerplumbing16.co.uk",
  "Phone Number": "[phone]", "URL": "https://pioneerplumbing16.co.uk#contact",
  "Full Address": "5449 Market St, London", "City": "London", "Country": "gb", "Industry": "Plumber" }
{ "name": "  Pioneer  Plumbing ", "emails": ["[e-mail]"], "phones": [""], "site": "",
  "street_address": "5449 Market St, London", "town": "London", "country": "gb", "type": "Plumber" }

The merged record keeps the valid email from the second row and the phone and website from the first.

The expected output for the 500-row file is easy to state because I built it: 500 minus 90 duplicates minus 10 junk rows is 400 records. That gives me something a script can assert on.

The caveat, up front: this is a synthetic file that I wrote, using assumptions I also hold when writing the matcher. A result of "zero errors" on it tells me the code is consistent and guards against regressions. It does not tell you how the tool behaves on your Google Maps export. Treat everything below as a development story, not a benchmark.

With that said, the final numbers on the fixture:

Measure	Result
Records returned	400
Duplicates merged	90
False merges	0
Missed duplicates	0
Run time on Apify	about 2 seconds

Getting to "zero false merges" is where the work was.

Bug 1: shared email addresses merged unrelated businesses

My first blocking keys were the obvious ones: website domain, normalized phone, email address, and name plus city. If two rows share a key, they are candidates for merging.

The email key was a mistake. Plenty of small businesses list a personal gmail.com address, and in a larger list several unrelated businesses will list the same one (a franchise owner, a freelancer who runs three storefronts, a bookkeeper). A few rows also had disposable-domain addresses. Keying on the full address merged all of them into one record.

The fix was to treat an email as a dedupe key only when it is a business address. Free-mail providers and disposable domains are excluded from the key, but the address is still cleaned, validated and kept on the record. The rule is now: a shared [e-mail] is evidence of one business; a shared [e-mail] is not.

Bug 2: "Red Rock Bakery" and "Red Rock Realty"

For fuzzy name matching I used Jaro-Winkler, which rewards a shared prefix. That is what you want for "Acme Plumbing" versus "Acme Plumbing Co", and exactly what you do not want here. Both names start with red rock, and once two such businesses sit in the same city, the similarity is high enough to merge them: Jaro-Winkler scores red rock bakery against red rock realty at 0.931, just above my 0.93 threshold.

The fix was a word-compatibility rule: after normalizing, if each name has a word the other one lacks, the pair is rejected no matter how high the similarity score is. "Bakery" is not in the other name, "Realty" is not in this one, so they stay separate. A name with only extra words on one side ("Acme Plumbing" and "Acme Plumbing Services") is still a candidate. The rule is small:

// words match if similar enough; any word containing a digit must match exactly
const wordsMatch = (a: string, b: string) =>
  /\d/.test(a) || /\d/.test(b) ? a === b : jaroWinkler(a, b) >= 0.85;

// reject a pair when each side has a word the other side lacks
function wordsCompatible(a: string, b: string): boolean {
  const lacking = (from: string[], against: string[]) =>
    from.some((w) => !against.some((o) => wordsMatch(w, o)));
  const wa = a.split(' ');
  const wb = b.split(' ');
  return !(lacking(wa, wb) && lacking(wb, wa));
}

The follow-up post in this series covers the rule and the similarity numbers in detail.

Bug 3: "Store 1000" and "Store 1001"

Chain businesses are the worst case for fuzzy matching. Branch names differ by one character, so the similarity is very high: Jaro-Winkler scores store 1000 against store 1001 at 0.96. I only caught this one because a 10,000-row synthetic run merged 5,465 records when I expected about 1,000. Two branches in the same city are two different places, and merging them throws away a real lead.

The fix is the digit check in the snippet above: numbers in a name must match exactly. If the digit sequences differ, the pair is distinct. This is crude and it is correct far more often than it is wrong. The cost is that a true duplicate can survive when two sources disagree on the number itself, and I would rather leave a duplicate than silently delete a branch.

Bug 4: the Actor could not read the dataset

This one is not about data quality. The input takes a dataset ID so the Actor can be chained after any scraper. It worked locally. The first run on the platform, which uses limited permissions, failed with a 403 because it could not read a dataset it had not created.

Actors running with limited permissions cannot read arbitrary datasets by ID. The platform expects access to be granted through the input. The fix was to declare the dataset input as a resource picker in the input schema, so the user picks the dataset in the Console and the run is granted access to it.
If you build any Actor that consumes another Actor's output, test it under limited permissions before you ship. It cost me a debugging session.

Bug 5: monorepo plus apify push

The logic lives in a shared packages/core package (kept free of Node-only APIs so it can also run in a Cloudflare Worker later), and the Actor lives in a sibling folder. apify push uploads the Actor folder and builds it on the platform. The platform build does not know about my workspace layout, so the shared package cannot be resolved from the Actor folder alone.

The fix was to bundle the Actor and its core code into a dist build before pushing, so the uploaded folder is self-contained. There was a second catch: apify push skips anything matched by .gitignore, and dist was ignored, so the first build died with Cannot find module '/usr/src/app/dist/main.js'. A !dist/ line in .actorignore forces it back in. It is a build chore, not a design problem, but it is the kind of thing that costs an hour the first time.

Scale and cost

The 500-row run is a correctness test. For scale I ran 10,000 records with live DNS checks on every email domain on Apify. That took about 61 seconds. (The email domains in that test were synthetic and mostly do not exist, so real lists can be faster or slower depending on how quickly the domains answer DNS.) The measured platform cost was about $0.005 per 1,000 records returned, so compute is a rounding error next to the price.

The price I set is $1.50 per 1,000 cleaned records, pay per event, and merged duplicates are not charged. On the fixture that means 400 records, which is about $0.60. I am telling you this so the arithmetic is checkable, not to argue it is cheap or expensive for your situation.

What I would do differently

1. Write the false-merge tests before the matcher. All three merge bugs were cases I could have listed on day one: shared inboxes, shared prefixes, numbered branches.
2. Make every merge explainable. Each output record carries a dupeGroupId and each score carries a reasons list, which made these bugs visible instead of silent.
3. Test on a messier fixture than you think you need. Mine had three schemas and it still did not cover everything real scrapers produce.

If you want to try it

The Actor is called Lead Data Cleaner, Deduper & Scorer. It cleans, dedupes and scores records from any Apify dataset or pasted JSON. It does not scrape, send email or probe mail servers, and it stores nothing after the run. The email check is a DNS lookup of the domain only.

It is live on Apify Store at https://apify.com/hiredhandai/lead-data-cleaner. Questions or bug reports: [e-mail]. I am the author, and I am also on X as @lbishopdev.

If you have a type of messy record that breaks it, I would like to see the (anonymized) shape of it.

Comments

[Muhammed Rasin O M]

Writing the generator first with a known expected output of 400 is what made the numbered-branch bug visible, and the 10k run is a good argument for scaling fixtures instead of hand-picking cases. One thing that might narrow the gap with real Google Maps exports: skew the duplicate and shared-email rates by category, since chains and franchises cluster far more than a flat 90 in 500. Did the numbered-branch rule hold up once you tried real chain data?

[오피셜메일]

공용 이메일을 곧바로 동일 사업자의 증거로 보지 않고, 숫자가 다른 지점을 별개로 남긴 사례가 인상적입니다. 자동 병합 근거와 원본 행을 함께 확인할 수 있다면, 경계 사례를 사람이 검토한 뒤 규칙에 반영하기도 수월하겠네요. 합성 데이터 결과의 한계를 먼저 밝혀 둔 점도 좋습니다.