The same website 5 metres apart is one pub, and the same name 430 metres apart needs a street number

Nakodo's business campaigns write to businesses that might buy a brand's product: cafes, gyms, salons, shops. Those businesses come from Overture Maps, an open places listing released every month, and our how it works page says so out loud, along with the 200 new businesses a day each campaign adds and the rule that treats a listed brand, or a website shared by more than three places, as a chain.

What the page did not account for until this week is that Overture sometimes holds one business twice. It is built from several sources, and when two of them describe the same pub with slightly different names, or place it 300 metres apart, the merge does not always happen. For us that is not a cosmetic problem. Two rows are two candidates, two websites read, and potentially two emails into the same inbox under the brand's name.

So src/lib/places/twins.ts now answers one question: are these two listings the same business?

Three keys, not a string comparison

// "The King's Head & Arms" and "Kings Head and Arms" are one name.
const nameKey = (name: string) =>
  name
    .toLowerCase()
    .replace(/&/g, " and ")
    .replace(/['’`]/g, "")
    .replace(/^\s*the\s+/, "")
    .replace(/[^\p{L}\p{N}]+/gu, "");

The name key drops punctuation entirely, so spacing and hyphens stop mattering, and it drops a leading "the", because half the pubs in Britain are listed both ways.

The address key is a word list with British street abbreviations folded one way:

const ABBREVIATIONS: Record<string, string> = { street: "st", road: "rd", lane: "ln", avenue: "ave", /* ... */ north: "n", south: "s" };

"63 Sancroft Street" and "63 Sancroft St." come out identical. The website key is the host with www. removed plus the path, and the path is the part that earns its keep later.

Distance decides which rule applies

Two constants:

const NEAR_KM = 0.1;
const SAME_ADDRESS_KM = 1; // how far apart two listings at one address can be placed

Over a kilometre apart, nothing is a twin. Within 100 metres, a shared name or a shared website is enough. In between, a shared name is only enough if the addresses agree.

The website rule has an exception that matters more than the rule:

if (a.site && b.site && a.site.host === b.site.host) {
  // Two pages of one site: a chain's (or a hotel's) for two of its places
  if (a.site.path !== b.site.path && a.site.path && b.site.path) return false;
  if (km <= NEAR_KM) return true;
}

A chain gives every pub a page on one domain. Two different paths on the same host are two businesses, and saying so costs one comparison. An empty path on one side is the home page standing in for a deeper page, which is the shape a duplicate actually takes.

And when the names match but the listings are hundreds of metres apart, the addresses have to carry it:

function sameAddress(a: Key, b: Key): boolean {
  if (!a.address || !b.address) return false;
  const [short, long] = a.address.length <= b.address.length ? [a.address, b.address] : [b.address, a.address];
  if (short !== long && !long.startsWith(`${short} `)) return false;
  if (a.postcode && b.postcode) return a.postcode === b.postcode;
  return /\d/.test(short);
}

One address is often the other with the town appended, hence the prefix test. Without two postcodes to compare, a street number is the only thing that makes an address specific: "North Rd" in two towns is two different Royal Oaks, and the digit test is what keeps them apart.

The tests read like a pub crawl, and all six pass:

✔ the same website a few metres apart is one business, whatever the names
✔ a chain's pages for two of its pubs are two businesses, even under one name
✔ the same name close by is one business, even with different websites
✔ the same name further off needs the same address
✔ one address with two names is two businesses
✔ the index finds an earlier twin across grid cells, and nothing else

The fifth one is the case that stops this being over-eager: Ku Bar and Bombshell share 30 Lisle Street and are two venues.

A grid, because a campaign holds thousands of places

The check is pairwise, and a campaign can hold thousands of businesses. So places go into a grid of 0.01 degree cells and only the neighbourhood is searched:

const dLng = Math.ceil(SAME_ADDRESS_KM / (KM_PER_DEGREE * Math.max(0.05, Math.cos((place.lat * Math.PI) / 180))) / CELL);

The cos(lat) correction is the same one our radius maths uses, with the same 0.05 floor so a campaign near a pole does not ask for an infinite number of columns. TwinIndex.find walks the cells within a kilometre and returns the first earlier place that matches, which is why one test deliberately straddles a cell edge.

Which listing wins

In the search itself, candidates are sorted by Overture's own confidence before the index is built, so the listing the data is surest of becomes the keeper and the other one loses. The loser is not simply dropped: it is written as a campaignChannels row already rejected, with the reason "Listed twice". A dropped row would come back in tomorrow's query and be re-examined for ever.

For campaigns that ran before any of this existed, there is a backfill verb, and the ordering is a different question there, because some of those listings have been written to:

const rank = (r: Row) => [Number(r.written), Number(r.stage !== "rejected"), r.confidence ?? 0, r.score ?? 0];
rows.sort((a, b) => {
  const [x, y] = [rank(a), rank(b)];
  const i = x.findIndex((v, k) => v !== y[k]);
  return i < 0 ? 0 : y[i] - x[i];
});

A tuple compared left to right: written to beats not written to, still live beats already set aside, then Overture's confidence, then the score. Whatever has already happened in the world outranks whatever the data prefers.

It is a dry run unless you ask otherwise:

$ pnpm pipeline twins
0 twins found (dry run; --apply sets them aside)

Zero is the correct answer now, and it is the whole idempotence story. The applied run set the duplicates to rejected, and the collector skips rows that are already rejected or already written to:

else if (!r.written && r.stage !== "rejected") twins.push({ row: r, keeper });

Run it twice and the second run has nothing to say, which is what you want from a verb an operator might paste into a terminal twice.

The guard that was looking at one address

There was a related hole upstream. Outreach already refused to start a second conversation with a creator found on two platforms, by comparing the address a thread was opened at against the addresses a candidate publishes. A twin breaks that: the two listings can publish overlapping sets of addresses while the one we actually used is not in the candidate's set.

So the guard now compares against every address the already-contacted listing publishes, in TypeScript:

const written = new Set([...threads.map((t) => t.emailHash), ...published.map((p) => emailHash(p.value))]);

and in the SQL that picks candidates, with a note for the next reader explaining why the subquery is deliberately uncorrelated:

-- (Not correlated, so the campaign's addresses are gathered once.)
not exists (
  select 1 from channel_contacts c
  where c.channel_id = campaign_channels.channel_id and c.kind = 'email'
    and lower(trim(c.value)) in (
      select lower(trim(tc.value)) from outreach_threads t
      join channel_contacts tc on tc.channel_id = t.channel_id and tc.kind = 'email'
      where t.campaign_id = $1
    )
)

Two identical-looking not exists blocks sitting next to each other is not duplication here. One asks "has anything we wrote to published this candidate's address", the other asks "does this candidate publish anything the things we wrote to publish". The twin case only trips the second.

If you want to see the product side of this, business outreach is a Pro and Business feature and the how it works page covers where the places come from and what counts as a chain. The counting rule on the pricing page is the reason any of this matters: a business counts once, when Nakodo writes to it, and a business written to twice would have been a brand's budget spent on annoying one cafe.