DEV Community

Cover image for Building a B2B Lead Enricher and a 2-Opt Route Optimizer in Python (No Headless Browser)
Trung Thanh Tran
Trung Thanh Tran

Posted on Edited on

Building a B2B Lead Enricher and a 2-Opt Route Optimizer in Python (No Headless Browser)

When sales operations and field-dispatch teams build outbound or territory pipelines, they almost always run into two surprisingly expensive bottlenecks:

  1. Dirty B2B Lead Lists Full of Corporate Chains & Dead Domains: You scrape 2,000 healthcare, dental, audiology, or veterinary clinics from a local map directory—only to realize that a good share of them are franchises or private-equity DSOs/MSOs (Aspen Dental, Heartland Dental, Miracle-Ear, Banfield), and that some of the scraped "emails" are sentry.wixpress.com telemetry strings or addresses on dead domains.
  2. Zig-Zag Field Territory Routes: Once you have 15–25 verified local prospects in a metro area, ordering visits chronologically or alphabetically can waste a big part of a rep's shift crisscrossing highways—while enterprise Route Matrix APIs charge per-element fees (N × N matrix billing).

Both problems can be handled in plain Python 3.11 with no headless browser and no paid third-party API keys (geocoding uses the free OpenStreetMap Nominatim service, which allows about 1 request per second).

In this deep dive, we will walk through the engineering behind three small Apify Actors I published recently (the source is public on GitHub; they are new and have only a handful of runs so far):


Stage 1: Zero-Browser B2B Contact Enrichment + DNS/MX Verification

Most naive email scrapers run a single regex ([a-zA-Z0-9._%+-]+@...) across raw homepage HTML and call it a day. On modern websites, that immediately pollutes your CRM with three classes of garbage:

  • Asset filename collisions: Retina image naming conventions like hero-banner@2x.png or logo@3x.webp.
  • Frontend error-tracking DSNs: Bundled Sentry keys such as 89f2a1@o4504.ingest.sentry.io or user@sentry-next.wixpress.com.
  • Unroutable / parked domains: Legacy domains in outdated footers with no valid MX or A records.

Step 1A: Async Domain MX Verification with In-Memory Caching

Instead of paying per-email SMTP verification credits upfront, you can drop dead domains cheaply (each domain is looked up once and cached) using asynchronous DNS MX lookups via dnspython's dns.asyncresolver, falling back cleanly to A record resolution per RFC 5321 Section 5.1:

import asyncio
import re
import dns.asyncresolver

EMAIL_REGEX = re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,12}")
JUNK_EXTS = (".png", ".jpg", ".jpeg", ".gif", ".svg", ".webp", ".css", ".js", ".woff", ".woff2")
JUNK_DOMAINS = {"sentry.io", "sentry.wixpress.com", "sentry-next.wixpress.com", "example.com", "wix.com"}


class AsyncMXValidator:
    def __init__(self) -> None:
        self._domain_cache: dict[str, bool] = {}

    async def has_valid_mail_route(self, domain: str) -> bool:
        d = domain.strip().lower().rstrip(".")
        if not d or "." not in d or d in JUNK_DOMAINS or "sentry" in d:
            return False
        if d in self._domain_cache:
            return self._domain_cache[d]

        resolver = dns.asyncresolver.Resolver()
        resolver.lifetime = 3.0
        deliverable = False
        try:
            mx_answers = await resolver.resolve(d, "MX")
            deliverable = len(mx_answers) > 0
        except Exception:
            # RFC 5321 fallback: implicit MX via A record if no explicit MX exists
            try:
                a_answers = await resolver.resolve(d, "A")
                deliverable = len(a_answers) > 0
            except Exception:
                deliverable = False

        self._domain_cache[d] = deliverable
        return deliverable


async def extract_verified_emails(html: str, validator: AsyncMXValidator) -> list[str]:
    candidates: set[str] = set()
    for raw in EMAIL_REGEX.findall(html):
        em = raw.strip().lower().rstrip(".,;:")
        if any(em.endswith(ext) for ext in JUNK_EXTS):
            continue
        local, _, domain = em.partition("@")
        if local and domain and len(local) <= 64:
            candidates.add(em)

    verified: list[str] = []
    for em in sorted(candidates):
        if await validator.has_valid_mail_route(em.split("@")[1]):
            verified.append(em)
    return verified
Enter fullscreen mode Exit fullscreen mode

Because B2B enrichment crawls /, /contact, and /about on the same domain, caching by domain means most candidate checks after the first page hit _domain_cache instead of the network. (An MX record only proves the domain can receive mail; it does not prove that a specific mailbox exists.)


Step 1B: Heuristic Corporate Chain / DSO / Franchise Screening

If you sell medical devices, dental lab services, hearing aids, or practice-management SaaS to independent clinic owners, calling a disguised corporate clinic is a complete waste of SDR dial time—local clinic managers cannot sign software or vendor contracts when procurement is locked at corporate HQ.

Even when a private-equity Dental Support Organization (DSO) or veterinary consolidator keeps a local-sounding brand name ("Oak Creek Family Dental"), they often leave footprints like these in the HTML and Schema.org metadata:

  1. Corporate Privacy Policy & Script Domains: Footer links pointing to heartland.com, aspendental.com, banfield.com, vcahospitals.com, miracle-ear.com, beltone.com, or thejoint.com.
  2. Schema.org parentOrganization / brand Declarations: Nested inside <script type="application/ld+json"> for MedicalOrganization, Dentist, or VeterinaryCare.
  3. Franchise / MSO Legal Disclaimers: Regex-detectable regulatory boilerplate in the footer ("each office is independently owned and operated", "backed by ... dental partners").
  4. Store-Locator URL Cardinality: Counting unique internal <a href="/locations/..."> subpaths (the Actor treats >= 15 distinct location slugs as a regional/national multi-site operator, and 5–14 as a regional group worth reviewing).

Each signal maps to a fixed score and the Actor keeps the highest one (a simple rule-based 0.0–1.0 score, not a trained model), then labels each enriched clinic:

  • Verified Independent (score < 0.45): Route straight into outbound CRM sequences.
  • Needs Review (0.45 <= score < 0.80): Ambiguous or multi-office group (e.g., 5–14 location pages, or a declared parentOrganization with no other chain signal), worth a manual look.
  • Chain Excluded (score >= 0.80): Automatically suppressed when excludeChainsFromOutput=True.

The default input of apify-us-healthcare-clinic-contact-enricher is a small mixed batch (pacificaudiology.com, brooklyndentalstudio.com, banfield.com). If the page loads, banfield.com matches a built-in brand signature and is labelled Chain Excluded with the signal Banfield Pet Hospital (Mars Veterinary Health). The other records get US phones normalized to +1-XXX-XXX-XXXX and the emails whose domains pass the MX check. Actual results depend on what each site serves on the day you run it.


Stage 2: Multi-Stop Route Ordering with Nearest-Neighbor + 2-Opt

Now suppose your enrichment pipeline outputs 18 verified independent clinics or property leads across Kansas City or Austin for a field representative who only has a 120-minute afternoon window (maxMinutes=120, with 12 minutes on-site dwell time per stop).

Why Road-Adjusted Haversine Is Often Good Enough for Clustering & Ordering

Calling a commercial Distance Matrix API on 25 stops requires 25 × 25 = 625 matrix elements per run. Across hundreds of daily territory batches, matrix billing adds up fast.

For clustering and ordering stops, you often don't need exact road distances. A common shortcut, studied in circuity-factor research (for example Ballou, Rahardja & Sakai, 2002), is to estimate road distance as great-circle Haversine distance multiplied by a circuity factor. The 1.28 default below is my own starting value, not a measured figure for any particular city, so calibrate it against a few real routes in your area:

import math


def road_adjusted_haversine_miles(
    lat1: float, lng1: float, lat2: float, lng2: float, circuity: float = 1.28
) -> float:
    """Computes street-grid adjusted distance in statute miles."""
    r_miles = 3958.7613
    p1, p2 = math.radians(lat1), math.radians(lat2)
    dp = math.radians(lat2 - lat1)
    dl = math.radians(lng2 - lng1)

    a = math.sin(dp / 2.0) ** 2 + math.cos(p1) * math.cos(p2) * math.sin(dl / 2.0) ** 2
    c = 2.0 * math.atan2(math.sqrt(a), math.sqrt(max(0.0, 1.0 - a)))
    return r_miles * c * circuity
Enter fullscreen mode Exit fullscreen mode

Why Nearest-Neighbor Alone Isn't Enough (And How 2-Opt Fixes It)

A greedy Nearest-Neighbor heuristic ("always drive to the closest unvisited stop") is fast (O(N²)), but it is notoriously myopic: early greedy hops leave isolated outlier stops behind, forcing long criss-crossing return legs at the end of the shift (on random instances, Johnson & McGeoch's TSP experiments put nearest-neighbor tours roughly 25% above optimal).

Adding 2-Opt Local Search removes most of those crossings. The simple version below recomputes the whole tour length for every candidate swap, so it slows down quickly as N grows (roughly O(N³) per pass): fine for 15–30 stops, but over a second for 60 stops on my laptop (numbers below).

The geometric intuition behind 2-Opt is simple: take any pair of non-adjacent edges in your tour, reverse the sub-segment tour[i : k + 1] between them, and check if the total route mileage strictly decreases. Repeat until no edge swap improves the tour:

Point = tuple[float, float]


def tour_miles(
    pts: list[Point],
    order: list[int],
    origin: Point,
    dest: Point | None,
    circuity: float = 1.28,
) -> float:
    if not order:
        return 0.0
    dist = road_adjusted_haversine_miles(*origin, *pts[order[0]], circuity)
    for idx in range(len(order) - 1):
        dist += road_adjusted_haversine_miles(*pts[order[idx]], *pts[order[idx + 1]], circuity)
    if dest is not None:
        dist += road_adjusted_haversine_miles(*pts[order[-1]], *dest, circuity)
    return dist


def optimize_stops_2opt(
    pts: list[Point],
    origin: Point,
    dest: Point | None = None,
    circuity: float = 1.28,
) -> list[int]:
    n = len(pts)
    if n <= 1:
        return list(range(n))

    # 1. Greedy Nearest-Neighbor seed tour from fixed origin
    unvisited = set(range(n))
    tour: list[int] = []
    curr = origin
    while unvisited:
        nxt = min(unvisited, key=lambda i: road_adjusted_haversine_miles(*curr, *pts[i], circuity))
        tour.append(nxt)
        unvisited.remove(nxt)
        curr = pts[nxt]

    # 2. Iterative 2-Opt edge-swap refinement (origin & optional dest remain anchored)
    best = tour_miles(pts, tour, origin, dest, circuity)
    improved = True
    while improved:
        improved = False
        for i in range(n - 1):
            for k in range(i + 1, n):
                candidate = tour[:i] + list(reversed(tour[i : k + 1])) + tour[k + 1 :]
                cand_dist = tour_miles(pts, candidate, origin, dest, circuity)
                if cand_dist + 1e-6 < best:
                    tour, best = candidate, cand_dist
                    improved = True
    return tour
Enter fullscreen mode Exit fullscreen mode

Benchmark: Random Order vs. Nearest-Neighbor vs. NN + 2-Opt

I ran the code above on my laptop (Python 3.11): for each size, 20 sets of random points inside a ~30 × 30 mile box around Kansas City, closed round trip from the center, circuity = 1.28. Values are medians. Distances are Haversine × 1.28 estimates, not real road miles, and a random input order is a deliberately bad baseline:

Stop Count (N) Random Input Order Nearest-Neighbor Only Nearest-Neighbor + 2-Opt Solver Time (median / max)
6 stops 140.4 mi 103.4 mi 97.6 mi (-5.6% vs NN) 0.33 ms / 0.67 ms
15 stops 303.6 mi 148.8 mi 138.0 mi (-7.2% vs NN) 6.0 ms / 9.8 ms
30 stops 600.9 mi 197.3 mi 185.0 mi (-6.3% vs NN) 55 ms / 98 ms
60 stops 1,192.3 mi 298.9 mi 256.0 mi (-14.3% vs NN) 1.28 s / 1.77 s

Most of the gain over a random order comes from nearest-neighbor; 2-opt then trims roughly another 5–15%. For larger N, precompute the distance matrix and score each swap by the change in its edge lengths instead of re-summing the whole tour. Also note that in the Actor itself, geocoding through Nominatim (about 1 request per second) takes far longer than the solver.

Once optimize_stops_2opt returns the improved visit order (2-opt finds a good local optimum, not a guaranteed optimal tour), enforcing a shift budget (maxMinutes = 120) is a single linear pass accumulating leg_drive_minutes + service_minutes—marking stops that fit inside the window as Scheduled and pushing overflow stops to Deferred for tomorrow's run, then URL-encoding the scheduled addresses directly into a 1-click mobile Google Maps Directions link:

https://www.google.com/maps/dir/?api=1&origin=...&destination=...&travelmode=driving&waypoints=Stop1|Stop2|Stop3
Enter fullscreen mode Exit fullscreen mode

Chaining Both Engines Together in n8n or Make.com

Because both engines follow a clean JSON-in / JSON-out contract, chaining them in n8n, Make.com, or Google Sheets takes four nodes:

  1. Ingest: Pull raw clinic or business URLs + street addresses from Google Sheets, HubSpot, or a Google Maps scraper.
  2. Enrich & Filter Chains: Call apify-us-healthcare-clinic-contact-enricher (or apify-b2b-lead-and-contact-enricher for general B2B tech-stack & contact enrichment) with excludeChainsFromOutput: true.
  3. Optimize Territory Route: Pass the surviving Verified Independent addresses into apify-multi-stop-route-and-distance-optimizer (for a simpler Haversine-only planner that runs inside Sheets, see the Apps Script example google_sheets_haversine_route_planner.js; it is written for one specific sheet layout and needs adapting).
  4. Dispatch: Push verified MX emails to your outreach tool and text the 1-click google_maps_nav_url itinerary straight to your field rep's phone.

All repositories linked above are public. The three Actors are also published on Apify Store (@thanh81646), so you can call them through Apify's standard synchronous /run-sync-get-dataset-items endpoint. They are new and have only a few runs so far, so treat them as worked examples rather than battle-tested tools. Feel free to clone, benchmark, or wire them into your own workflows, and issues are welcome!


I'm Trung Thanh Tran, a Python developer building small data-enrichment and route-planning tools as Apify Actors and scripts.

  • Want 6 complete Python 3.11 CLI engines + 3 importable n8n/Make workflow blueprints in one instant download? Grab the Python, AI & n8n Scraper Toolkit Pro on Gumroad.
  • Looking for a custom B2B lead enrichment engine, Google Sheets / Apps Script route planner, or Apify Actor? Hire me on Upwork or Fiverr.

Top comments (0)