DEV Community

Gio Rich
Gio Rich

Posted on

Getting Better Business Bureau (BBB) business listings as JSON with Python (ratings, accreditation, phones)

#webscraping #python #leadgeneration #api

I built this actor; it's a paid tool on Apify with a free trial credit.

The Better Business Bureau lists millions of businesses across the US and Canada, and every listing in its search carries a few things that are hard to get elsewhere in one place: a BBB letter rating from A+ to F, whether the business is BBB Accredited, a phone number, a street address and a category. For anyone building local lead lists (roofers in Houston, dentists in Toronto, HVAC companies in Phoenix) that is a clean starting point, and the rating is a quick signal of who is worth calling.

BBB has no public self-serve API. I'm an 18-year-old engineering student, and I built BBB Business Scraper to turn bbb.org searches into JSON. This post covers calling it from Python and Node, what a row looks like, how it gets past BBB's 225-result limit, and a small project: a weekly list of accredited, A-rated contractors.

How it works

The search page on bbb.org loads its results from a JSON endpoint, 15 businesses per page. The actor calls that same endpoint over plain HTTP, no browser, and keeps the business fields: name, category, rating, accreditation, phones, address, coordinates, the profile URL, the local BBB and the service area. BBB highlights the matched word with <em> tags in names, so those are stripped.

The catch is that BBB never returns more than 15 pages for one search, so 225 businesses, even when it says there are 1,954 plumbers around Denver. Page 16 doesn't error, it just hangs until Cloudflare gives up. So when you ask for more than 225, the actor runs narrower searches and merges them by id: one per rating grade (A, B, C, D, F), then per state BBB offers for that search, then per category, then per distance ring (5, 10, 25, 50 miles). A slice that is still too big is read in BBB's other sort orders (A-Z, Z-A, rating, distance each show a different 225), and as a last step the same slice is searched around the towns its businesses are in. Deeper slices are paged one page at a time and stop once pages stop bringing anything new. For plumber near Denver that reached 1,524 of the 1,954 BBB lists, in about 4 minutes and 1,200 requests.

Profile pages (complaints, reviews, owner names) are behind a Cloudflare bot check, so the actor doesn't touch them. Everything comes from the public search results.

Python example 

pip install apify-client
Enter fullscreen mode Exit fullscreen mode 
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")

run = client.actor("rel8ble/bbb-business-scraper").call(run_input={
    "searchQueries": ["roofing"],
    "locations": ["Austin, TX"],
    "accreditedOnly": True,
    "ratings": ["A"],
    "maxDistanceMiles": 25,
    "maxItems": 60,
})

for b in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(b["name"], "|", b["rating"], "|", b["phone"], "|", b["city"], "|", b["profileUrl"])
Enter fullscreen mode Exit fullscreen mode

In my test run every one of the 60 rows came back accredited and rated A+, all in Austin.

Node example 

npm install apify-client
Enter fullscreen mode Exit fullscreen mode 
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });

const run = await client.actor('rel8ble/bbb-business-scraper').call({
    searchQueries: ['dentist', 'plumber'],
    locations: ['Vancouver, BC', 'M5V 2T6'],
    maxItems: 30,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.length, 'businesses');
console.log(items[0].name, '-', items[0].phone, '-', items[0].localBbb);
Enter fullscreen mode Exit fullscreen mode

What a row looks like 

{
    "id": "1296_90156205_690834",
    "businessId": "90156205",
    "name": "Ennis Plumbing LLC",
    "category": "Plumber",
    "categories": ["Plumber", "Commercial Plumber"],
    "rating": "A+",
    "ratingScore": 100,
    "accredited": true,
    "phone": "(720) 660-9812",
    "address": "4865 Golden Ct",
    "city": "Denver",
    "state": "CO",
    "postalCode": "80212",
    "country": "US",
    "latitude": 39.74,
    "longitude": -104.97,
    "profileUrl": "https://www.bbb.org/us/co/denver/profile/plumber/ennis-plumbing-llc-1296-90156205",
    "serviceAreas": ["Thornton, CO"],
    "localBbb": "BBB Great West + Pacific"
}
Enter fullscreen mode Exit fullscreen mode

(The full row has a few more fields: all phone numbers, logo, review and quote links, the search it came from.) id is one business location; a company with several branches has one businessId and several rows.

Small project: accredited, A-rated contractors every week

If you sell to home-service businesses, the accredited A-rated ones are usually established and reachable. This script runs weekly, searches a few trades in a few cities, keeps only businesses it hasn't seen before and prints one line each.

import json, os
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
SEEN = "seen_bbb.json"
seen = set(json.load(open(SEEN))) if os.path.exists(SEEN) else set()

run = client.actor("rel8ble/bbb-business-scraper").call(run_input={
    "searchQueries": ["roofing contractors", "hvac", "electrician"],
    "locations": ["Houston, TX", "Dallas, TX", "Austin, TX"],
    "accreditedOnly": True,
    "ratings": ["A"],
    "maxDistanceMiles": 25,
    "maxItems": 0,
})

new = []
for b in client.dataset(run["defaultDatasetId"]).iterate_items():
    if b["businessId"] in seen or not b["phone"]:
        continue
    seen.add(b["businessId"])
    new.append(b)

for b in sorted(new, key=lambda x: (x["city"] or "", x["name"] or "")):
    print(f'{b["name"]} | {b["category"]} | {b["phone"]} | {b["city"]}, {b["state"]} | {b["profileUrl"]}')

json.dump(sorted(seen), open(SEEN, "w"))
print(len(new), "new businesses")
Enter fullscreen mode Exit fullscreen mode

Deduplicating on businessId rather than id gives one line per company even when it has several branches. Put it on cron or an Apify schedule and push the lines into a CRM or a sheet.

Things to know

  • Price: $1.00 per 1,000 businesses, no start fee. 100 businesses is 10 cents.
  • The 225 limit is BBB's. Deep search is on by default and only kicks in when you ask for more than 225 from one search; the run summary shows BBB's count, what was reached and the coverage percent. Expect most, not all, of a very big search.
  • Wide area: a city search reaches far (Denver returns Wyoming and Nebraska too). Use maxDistanceMiles or states to tighten it.
  • Typos in places: BBB answers an unknown place by searching everything. The actor spots that, skips the search and doesn't charge for it.
  • No profiles, no personal data: complaints, reviews and owner names live on profile pages behind a bot check, which the actor doesn't open. E-mail addresses that look personal in free text are masked.
  • US and Canada: "City, ST", ZIP codes, Canadian postal codes and state or province codes all work.

Questions or broken fields: the Issues tab on the actor page comes straight to me.

BBB Business Scraper on Apify

This article was drafted with AI and published by me.

Top comments (0)