DEV Community: tmeteor-data

Extracting business emails and phones from websites with Python

tmeteor-data — Sat, 24 Oct 2026 09:07:21 +0000

I built an Apify actor called Website Contact Extractor, so treat my opinions with that in mind. An AI assistant helped draft this article and I reviewed and edited it. The free script comes first because for a few sites it is enough.

The problem

You have a list of business websites and need a way to reach them: a published email, a phone number, a social profile. Doing it by hand means opening the homepage, hunting for a Contact link, and copying values into a sheet. That scales to about twenty sites before it hurts.

The free route

Businesses that want to be contacted usually publish details in predictable places: mailto: and tel: links on the homepage and on a /contact page, and sometimes structured data in a JSON-LD block (schema.org Organization or LocalBusiness). The script below checks all three, and honors robots.txt before fetching.

import json, re, requests
from urllib.parse import urljoin
from urllib.robotparser import RobotFileParser

UA = "contact-check/0.1 (you@example.com)"

def contacts(base):
    rp = RobotFileParser(urljoin(base, "/robots.txt"))
    rp.read()
    out = {"emails": set(), "phones": set(), "jsonld": []}
    for path in ("/", "/contact"):
        url = urljoin(base, path)
        if not rp.can_fetch(UA, url):
            continue
        r = requests.get(url, timeout=15, headers={"User-Agent": UA})
        if r.status_code != 200:
            continue
        html = r.text
        out["emails"] |= {m.split("?")[0] for m in
                          re.findall(r'href=["\']mailto:([^"\']+)', html, re.I)}
        out["phones"] |= set(re.findall(r'href=["\']tel:([^"\']+)', html, re.I))
        for blk in re.findall(r'<script[^>]+ld\+json[^>]*>(.*?)</script>',
                              html, re.S | re.I):
            try:
                out["jsonld"].append(json.loads(blk))
            except ValueError:
                pass
    return out

print(contacts("https://example.com"))

Politeness matters more than cleverness here:

Read robots.txt (the script does) and skip what it disallows.
Identify yourself in the User-Agent, with a way to reach you.
Fetch one page at a time per site, with a delay when you loop over many pages.
Only collect what the business published for contact purposes. A named person's email on a team page can be personal data under rules like GDPR, so think about what you store and why. This is a general caution, not legal advice.

What the free route does not do

The script finds the easy cases and nothing more:

Finding the right pages. /contact is a guess. Real sites use /contact-us, /impressum, /about, /locations or a footer link with some other name. You need link-text scoring to find them.
Phone formats. tel: values come as (313) 555-0142, 313.555.0142 or +1 313.... Turning them into one comparable format needs a phone-number library and a country assumption.
Obfuscated and text emails. Addresses written as info [at] example.com, or just shown as text rather than a mailto: link, are missed.
Role versus named. info@ and jane.doe@ are different kinds of lead, and the script does not tell them apart.
Social profiles and address. You would need more regexes and JSON-LD parsing for each.
Dead sites. Parked domains, DNS failures, bot challenges and timeouts need handling and reporting, or your results silently go missing.
Change monitoring. Knowing when a site's published contacts changed means keeping state between runs.

The actor

The actor wraps those problems into one run. You give it websites or bare domains and get one row per site. It fetches the homepage over plain HTTP (no browser), then the best contact, impressum, about, team and privacy pages it finds by link text and URL, up to a configurable cap. It respects robots.txt, fetches pages within one site one after another, and does not use logins or captcha bypass.

Example input:

{
  "startUrls": [{ "url": "https://buddyspizza.com" }],
  "maxPagesPerSite": 5,
  "defaultCountry": "US",
  "includeNamedEmails": true,
  "onlyWithContacts": false,
  "maxSites": 100
}

Output excerpt from a real run (fields trimmed; the full row also has per-source detail and a flat copy of each social field):

{
  "domain": "buddyspizza.com",
  "status": "ok",
  "companyName": "Buddy's Pizza",
  "companyNameSource": "schema.org",
  "emails": [
    { "email": "buddyspizza@buddyspizza.com", "type": "role",
      "foundOn": "https://buddyspizza.com/privacy-policy/",
      "pageType": "privacy", "method": "mailto" }
  ],
  "primaryEmail": "buddyspizza@buddyspizza.com",
  "phones": [
    { "phone": "+18009650505", "national": "(800) 965-0505", "pageType": "home", "method": "tel-link" },
    { "phone": "+18334510774", "national": "(833) 451-0774", "pageType": "contact", "method": "tel-link" }
  ],
  "socials": {
    "facebook": "https://www.facebook.com/buddyspizza",
    "instagram": "https://www.instagram.com/buddyspizzadet",
    "x": "https://x.com/buddyspizza"
  },
  "contactPageUrl": "https://buddyspizza.com/contact/",
  "pagesCrawled": 4
}

Notice where the email came from: the privacy page, not the homepage. That is the kind of thing a two-URL script misses. Each email and phone records the page it was found on and the method (mailto link, tel link and so on), so you can verify any value.

Other options: monitorMode remembers each site's emails, phones and socials between runs and returns only sites that changed; urlIncludePatterns and urlExcludePatterns steer the crawl; sameDomainOnly is on by default.

Limits and pricing

It finds only contacts the site publishes. It does not guess addresses and it does not verify that an email is deliverable.
A site that loads but publishes no email is still charged, because it was crawled. The row will still show whatever phones or socials exist.
Dead, blocked, parked and robots-disallowed sites come back with a status and are not charged.
Pricing is pay per event: $0.002 per site crawled plus $0.00005 per run start, so 1,000 sites is about $2.00 whether it crawls one page or five. Volume tiers are scheduled for paid Apify plans; check the Pricing tab for the current figure.
Named personal emails can be personal data. Use the output in line with the laws that apply to you.

Actor page: https://apify.com/transparent_meteorite/website-contact-extractor

Disclosure again: I'm the builder of this actor, and an AI assistant helped draft this article; I reviewed it before publishing.

Finding local businesses with no website using OpenStreetMap

tmeteor-data — Sat, 24 Oct 2026 09:07:21 +0000

I built an Apify actor that turns OpenStreetMap data into local business lead lists, so I have a stake here. An AI assistant helped me draft this article, and I reviewed and edited it. The free route is first because the Overpass API is genuinely good for this.

The problem

Web designers and local marketers want a simple list: dentists, plumbers or restaurants in a given city that have no website. Google Maps scraping is fragile and against its terms. OpenStreetMap (OSM) is volunteer-built open data with phone numbers, addresses and, often, a website tag. A business whose entry has no website tag is a candidate for a pitch.

The free route

Overpass is the read-only query API for OSM. This query finds named dentists inside Austin's city boundary that have no website tag.

import requests

QUERY = """
[out:json][timeout:60];
area["name"="Austin"]["boundary"="administrative"]["admin_level"="8"]->.a;
nwr["amenity"="dentist"]["name"][!"website"][!"contact:website"](area.a);
out center tags 50;
"""
resp = requests.post(
    "https://overpass-api.de/api/interpreter",
    data={"data": QUERY},
    headers={"User-Agent": "lead-list-example/0.1 (contact: your-real-email-here)"},
    timeout=90,
)
resp.raise_for_status()
for el in resp.json()["elements"]:
    t = el["tags"]
    print(t["name"], t.get("phone"), t.get("addr:street"), el["type"], el["id"])

Two practical notes. Overpass returned HTTP 406 to my requests until I sent a descriptive User-Agent with a real contact address (a placeholder address also failed), so replace the example string with your own details. Second, [!"website"] means the tag is absent, and the contact:website tag is a common alternative, which is why the query excludes both. Other tags a mapper might use for the same thing exist, so expect some noise. The public servers are shared and sometimes busy; keep queries small and retry politely. Your area name can match more than one place, so check the result against a map.

What the free route does not do

Telling "no site" from "not mapped". An absent tag usually means no site, but a mapper may simply not have added it. Verify before you pitch.
Locations as people type them. "Austin, TX" or a ZIP code needs geocoding and boundary lookup. The query above hard-codes a city name.
Plain-word categories. "Plumber" or "hair salon" maps to OSM tags you have to look up.
Flat output. Raw OSM elements carry assorted tags, so you must flatten address, hours and social links into columns.
Email enrichment. OSM rarely has emails. Getting them means fetching each business's own site, which you have to build and keep polite.
Deduplication. A business mapped as both a point and a building shows up twice.
Hosting. Retries across Overpass servers, schedules and exports are yours to run.

The actor

The actor takes plain-word business types and places, resolves official boundaries, queries OSM data, de-duplicates, and returns flat rows. With enrichFromWebsite on, it reads emails, phones and social links published on each business's homepage and contact page. It does not guess emails.

Input using the real field names (this combination asks for the no-website call list):

{
  "categories": ["dentist"],
  "locations": ["Austin, TX"],
  "maxItems": 10,
  "onlyWithoutWebsite": true,
  "requirePhone": true
}

Other real fields include requireWebsite, requireEmail, enrichFromWebsite, bbox, and latitude, longitude and radiusKm for circle searches.

Excerpt from the sample output file. That sample came from a plain dentists-in-Austin run, not the no-website filter, so these rows have websites:

{
  "name": "Central Texas Pediatric Dentistry",
  "category": "dentist",
  "phone": "+1-512-474-7575",
  "website": "https://www.ctpdkid.com/",
  "email": "ctpd@ctpdkid.com",
  "city": "Austin",
  "state": "TX",
  "osmUrl": "https://www.openstreetmap.org/node/2992260019",
  "source": "OpenStreetMap contributors, ODbL 1.0"
}

The sample file also has openingHours, socials, lat, lon and lastEdited on each row, and I trimmed the rest here.

Honest limits and pricing

Coverage depends on volunteers. The README says it is often good for shops, restaurants, clinics and dentists in big cities, and weaker for home-based trades and small towns. There are no reviews, ratings or photos.
A no-website result is a lead to check, not a fact.
Public Overpass servers can be busy; large areas can take longer.
Website enrichment reads at most two pages per site and does not run JavaScript.
Data is under the ODbL licence. Credit "OpenStreetMap contributors" and note the share-alike terms if you publish a database built from it. Each row's source field carries the attribution.
Pricing is pay per result: $0.003 per business ($3 per 1,000), plus $0.002 per website enriched if you turn that on, plus a tiny per-run start fee. The store listing mentions lower per-place prices on higher Apify plans, so check the page. Filtered-out places are not charged.

If you can live with raw OSM tags, the script costs nothing. If you want boundaries, filters and enrichment handled, the actor is here: https://apify.com/transparent_meteorite/local-business-leads

Disclosure again: I built the actor, and an AI assistant helped draft this article. I reviewed it and ran the Overpass snippet with a real contact address before writing this up.

Pulling open jobs from Greenhouse, Lever and Ashby without scraping HTML

tmeteor-data — Sat, 24 Oct 2026 01:32:36 +0000

I built an Apify actor that reads job boards from several applicant tracking systems (ATS), so keep that bias in mind. An AI assistant helped me draft this article, and I reviewed and edited it. The free route comes first, because for a handful of companies a short script is enough.

The problem

Recruiters, job-board builders and sales teams that watch hiring signals all want the same thing: a clean list of open roles at specific companies. Career pages are often JavaScript-heavy, but most companies run their hiring on an ATS that exposes a public JSON feed so the career page can render. You do not need to scrape HTML. You need the company's board slug, which is the last part of the careers URL (for example boards.greenhouse.io/airbnb, jobs.lever.co/spotify, jobs.ashbyhq.com/ramp).

The free route

import requests

def greenhouse(slug):
    r = requests.get(f"https://boards-api.greenhouse.io/v1/boards/{slug}/jobs", timeout=30)
    r.raise_for_status()
    return [(j["title"], j["location"]["name"], j["absolute_url"]) for j in r.json()["jobs"]]

def lever(slug):
    r = requests.get(f"https://api.lever.co/v0/postings/{slug}", params={"mode": "json"}, timeout=30)
    r.raise_for_status()
    return [(j["text"], j["categories"].get("location"), j["hostedUrl"]) for j in r.json()]

def ashby(slug):
    r = requests.get(f"https://api.ashbyhq.com/posting-api/job-board/{slug}", timeout=30)
    r.raise_for_status()
    return [(j["title"], j["location"], j["jobUrl"]) for j in r.json()["jobs"]]

for name, fn, slug in [("greenhouse", greenhouse, "airbnb"),
                       ("lever", lever, "spotify"),
                       ("ashby", ashby, "ramp")]:
    jobs = fn(slug)
    print(name, slug, len(jobs), jobs[0])

I checked these against the live endpoints. Greenhouse returns jobs with title, location.name, absolute_url, updated_at and first_published. Lever returns a list with text, categories (including location, department, commitment), hostedUrl, applyUrl and createdAt. Ashby returns jobs with title, department, location, isRemote, workplaceType, publishedAt and jobUrl. Field sets differ between vendors and can change, so check each response's keys before depending on one. For descriptions, Greenhouse needs ?content=true on the request.

What the free route does not do

A shared schema. Look at the three functions above: each uses different keys for title, location and date. Remote flags, employment type, department and salary live in different places, or nowhere. Mapping them into one shape is the real work.
Salary and seniority. Some boards publish pay ranges, many do not. Seniority has to be inferred from titles.
Finding the slug. You need the right board name for each company, and two companies can share a slug.
Only-new tracking. To get "jobs posted since yesterday" you must store what you saw and diff it, including changed and removed jobs.
Other vendors. Workable, SmartRecruiters and Recruitee each have their own formats, and some companies only publish schema.org JobPosting markup on their pages.
Hosting. A schedule, retries and a place to keep state.

The actor

The actor covers Greenhouse, Lever, Ashby, Workable, SmartRecruiters and Recruitee, plus a schema.org JobPosting fallback for single career pages. You can paste company slugs or careers URLs. The input below is the README's real test input:

{
  "companies": [
    "airbnb",
    "https://jobs.lever.co/spotify",
    "https://jobs.ashbyhq.com/ramp"
  ],
  "onlyNewSinceLastRun": false,
  "maxJobsPerCompany": 3,
  "maxJobsTotal": 3,
  "includeDescription": false
}

Other real fields include keywords, excludeKeywords, locations, departments, employmentTypes, remoteOnly, workModels, postedAfter, minSalary, skillsInclude and monitorMode (return only new or changed jobs, optionally also removed ones).

Excerpt of real sample output (description removed, a few fields kept):

{
  "company": "Spotify",
  "ats": "lever",
  "title": "Senior Product Manager, Consumer - Audiobooks",
  "department": "Product",
  "location": "London",
  "remote": false,
  "workplaceType": "hybrid",
  "employmentType": "FULL_TIME",
  "postedAt": "2026-10-12T13:42:44.230Z",
  "jobUrl": "https://jobs.lever.co/spotify/ddc10181-6ecc-4dfc-9d02-2c5ce820cfdd",
  "salaryMin": null,
  "seniority": "senior"
}

The Greenhouse row in the same file shows the vendor differences: Airbnb's "Senior Software Engineer, Payments Transaction Risk" has remote, workplaceType and employmentType all null, because Greenhouse's board feed does not carry them.

Honest limits and pricing

Six ATS vendors plus schema.org. Workday, iCIMS, Taleo, BambooHR, Personio and JavaScript-only career sites are not supported. An unsupported page returns zero rows and costs nothing.
Fields depend on the vendor. Greenhouse has no remote flag (the actor sets remote: true only when the location says remote), Lever has no updatedAt, and most boards publish no salary.
seniority is a heuristic from the title only. Treat it as a best guess.
No recruiter names or emails are collected.
Pricing is pay per result: $0.002 per job ($2.00 per 1,000) plus a $0.00005 start fee per run. Jobs removed by your filters, or already delivered when dedupe is on, are not charged. The store listing mentions lower per-job prices on higher Apify plans, so check the page for your plan.

For a few boards, run the script. For a watchlist with filters and a daily new-jobs feed, the actor is here: https://apify.com/transparent_meteorite/ats-open-roles

Disclosure again: I built the actor, and an AI assistant helped draft this article. I reviewed it and checked the three endpoints before writing the snippet.

Finding newly registered trucking companies from the FMCSA census

tmeteor-data — Fri, 23 Oct 2026 23:02:28 +0000

I built an Apify actor that turns FMCSA's public carrier census into a daily list of new trucking companies, so I'm biased. I also used an AI assistant to help draft this article, and I reviewed and edited it. The free route comes first because for many people it is all you need.

The problem

Anyone who sells to trucking startups (insurance, factoring, fuel cards, ELD and dispatch software, recruiting) wants one list: companies that registered with the Federal Motor Carrier Safety Administration in the last few days. Those companies are choosing vendors right now. FMCSA publishes the census as open data, but the raw file is a wide table with a text date column and no "what is new since yesterday" view.

The free route

The census is dataset az4n-8mr2 on data.transportation.gov, which runs Socrata, so you can query it with SoQL over plain HTTP. No key is needed for light use.

import requests
from datetime import date, timedelta

URL = "https://data.transportation.gov/resource/az4n-8mr2.json"
since = (date.today() - timedelta(days=7)).strftime("%Y%m%d")
params = {
    "$where": f"status_code='A' AND add_date >= '{since}' AND phy_state='TX'",
    "$order": "add_date DESC",
    "$limit": 50,
}
resp = requests.get(URL, params=params, timeout=60)
resp.raise_for_status()
for r in resp.json():
    print(r["add_date"], r["dot_number"], r["legal_name"],
          r.get("power_units"), r.get("email_address"))

Notes from running it against the live dataset: add_date is a text value in YYYYMMDD form, so a string comparison works for date windows. status_code='A' keeps active records. Other columns I saw include phy_state, legal_name, power_units, total_drivers, phone and email_address. Column sets can change, so check the dataset's column list before relying on a name. Socrata caps a single response, so page with $offset if you need more than a few thousand rows.

That is a perfectly good weekly script. Put it in cron, write the rows to a CSV, and you are done.

What the free route does not do

These are the gaps that made me build something instead of keeping a script:

Cross-run dedup. Your script returns the same carriers every time the window overlaps. You need your own store of USDOT numbers you have already seen.
Authority status. The census file does not tell you whether a carrier's operating authority is pending or granted. That lives in a separate FMCSA authority dataset that you would have to join yourself (the actor uses yu5v-wbh6 for this).
Normalized fields. The raw census uses codes such as A for carrier operation. The actor adds labels and ISO dates.
Filters people ask for. Fleet size, hazmat, cargo types, for-hire only (USDOT numbers are also issued to contractors and landscapers who move their own goods).
Scheduling and delivery. Hosting a cron job, retrying, and pushing rows to a webhook or CRM is your problem.

The actor

The actor reads the same federal dataset, joins the authority history, filters, remembers what it has already delivered, and exports JSON, CSV or Excel. It needs no API key besides your Apify account.

Example input (these are real input fields):

{
  "states": ["TX"],
  "daysBack": 2,
  "requireEmail": true,
  "onlyNewSinceLastRun": true,
  "maxResults": 100
}

A daily schedule with daysBack: 2 overlaps the previous day on purpose, so a late FMCSA update is not missed, while onlyNewSinceLastRun stops repeats.

Excerpt of real output from the sample file (contact fields trimmed here; they are in the actual rows):

{
  "legal_name": "RVRR TRANSPORT LLC",
  "usdot": "9912039",
  "state": "MN",
  "power_units": 1,
  "total_drivers": 4,
  "carrier_operation_label": "Interstate",
  "registered_date_iso": "2026-10-12",
  "census_status": "Active",
  "officer_1": "RAYMOND UDE",
  "authority_status": "Pending",
  "authority_status_reason": "Published to FMCSA Register"
}

In that sample file, the rows I looked at were one- or two-truck operators with authority still Pending, which is typical for brand-new registrations.

Honest limits and pricing

US carriers only, and the data is only as fresh as FMCSA's census, which is updated on FMCSA's schedule.
Email and phone are whatever the carrier filed. Many are personal inboxes or cell numbers, and carriers that filed no email have none. Follow TCPA and CAN-SPAM rules when you use them.
The census file has no insurance filing status.
Pricing is pay per result with no subscription, billed per unique new carrier delivered. As I write this (October 23, 2026) the event price is $0.008 per carrier, and I have a price change scheduled for early November, so check the store page for the price in effect before a big run. Apify's free plan credit covers a small test.

If you only need a weekly list for one state, use the script above. If you want a deduped daily feed without maintaining it, the actor is here: https://apify.com/transparent_meteorite/fmcsa-new-carrier-leads

Disclosure again: I built this actor, and an AI assistant helped draft this article. I reviewed the content and ran the free-route snippet against the live dataset.