Finding local businesses with no website using OpenStreetMap

#python #openstreetmap #api #abotwrotethis

I built an Apify actor that turns OpenStreetMap data into local business lead lists, so I have a stake here. An AI assistant helped me draft this article, and I reviewed and edited it. The free route is first because the Overpass API is genuinely good for this.

The problem

Web designers and local marketers want a simple list: dentists, plumbers or restaurants in a given city that have no website. Google Maps scraping is fragile and against its terms. OpenStreetMap (OSM) is volunteer-built open data with phone numbers, addresses and, often, a website tag. A business whose entry has no website tag is a candidate for a pitch.

The free route

Overpass is the read-only query API for OSM. This query finds named dentists inside Austin's city boundary that have no website tag.

import requests

QUERY = """
[out:json][timeout:60];
area["name"="Austin"]["boundary"="administrative"]["admin_level"="8"]->.a;
nwr["amenity"="dentist"]["name"][!"website"][!"contact:website"](area.a);
out center tags 50;
"""
resp = requests.post(
    "https://overpass-api.de/api/interpreter",
    data={"data": QUERY},
    headers={"User-Agent": "lead-list-example/0.1 (contact: your-real-email-here)"},
    timeout=90,
)
resp.raise_for_status()
for el in resp.json()["elements"]:
    t = el["tags"]
    print(t["name"], t.get("phone"), t.get("addr:street"), el["type"], el["id"])

Two practical notes. Overpass returned HTTP 406 to my requests until I sent a descriptive User-Agent with a real contact address (a placeholder address also failed), so replace the example string with your own details. Second, [!"website"] means the tag is absent, and the contact:website tag is a common alternative, which is why the query excludes both. Other tags a mapper might use for the same thing exist, so expect some noise. The public servers are shared and sometimes busy; keep queries small and retry politely. Your area name can match more than one place, so check the result against a map.

What the free route does not do

Telling "no site" from "not mapped". An absent tag usually means no site, but a mapper may simply not have added it. Verify before you pitch.
Locations as people type them. "Austin, TX" or a ZIP code needs geocoding and boundary lookup. The query above hard-codes a city name.
Plain-word categories. "Plumber" or "hair salon" maps to OSM tags you have to look up.
Flat output. Raw OSM elements carry assorted tags, so you must flatten address, hours and social links into columns.
Email enrichment. OSM rarely has emails. Getting them means fetching each business's own site, which you have to build and keep polite.
Deduplication. A business mapped as both a point and a building shows up twice.
Hosting. Retries across Overpass servers, schedules and exports are yours to run.

The actor

The actor takes plain-word business types and places, resolves official boundaries, queries OSM data, de-duplicates, and returns flat rows. With enrichFromWebsite on, it reads emails, phones and social links published on each business's homepage and contact page. It does not guess emails.

Input using the real field names (this combination asks for the no-website call list):

{
  "categories": ["dentist"],
  "locations": ["Austin, TX"],
  "maxItems": 10,
  "onlyWithoutWebsite": true,
  "requirePhone": true
}

Other real fields include requireWebsite, requireEmail, enrichFromWebsite, bbox, and latitude, longitude and radiusKm for circle searches.

Excerpt from the sample output file. That sample came from a plain dentists-in-Austin run, not the no-website filter, so these rows have websites:

{
  "name": "Central Texas Pediatric Dentistry",
  "category": "dentist",
  "phone": "+1-512-474-7575",
  "website": "https://www.ctpdkid.com/",
  "email": "ctpd@ctpdkid.com",
  "city": "Austin",
  "state": "TX",
  "osmUrl": "https://www.openstreetmap.org/node/2992260019",
  "source": "OpenStreetMap contributors, ODbL 1.0"
}

The sample file also has openingHours, socials, lat, lon and lastEdited on each row, and I trimmed the rest here.

Honest limits and pricing

Coverage depends on volunteers. The README says it is often good for shops, restaurants, clinics and dentists in big cities, and weaker for home-based trades and small towns. There are no reviews, ratings or photos.
A no-website result is a lead to check, not a fact.
Public Overpass servers can be busy; large areas can take longer.
Website enrichment reads at most two pages per site and does not run JavaScript.
Data is under the ODbL licence. Credit "OpenStreetMap contributors" and note the share-alike terms if you publish a database built from it. Each row's source field carries the attribution.
Pricing is pay per result: $0.003 per business ($3 per 1,000), plus $0.002 per website enriched if you turn that on, plus a tiny per-run start fee. The store listing mentions lower per-place prices on higher Apify plans, so check the page. Filtered-out places are not charged.

If you can live with raw OSM tags, the script costs nothing. If you want boundaries, filters and enrichment handled, the actor is here: https://apify.com/transparent_meteorite/local-business-leads

Disclosure again: I built the actor, and an AI assistant helped draft this article. I reviewed it and ran the Overpass snippet with a real contact address before writing this up.

DEV Community