BBB.org's business data isn't in the HTML — it's in an analytics script tag

Quick answer

BBB.org doesn't render business data into the HTML table it looks like it does. Both the search-results page and every business profile page embed the real data as one inline JSON blob — webDigitalData — sitting inside a <script> tag meant for analytics, not for you. The visible <dt>/<dd> list underneath it only carries the fields the analytics layer left out (accreditation date, years in business). If you scrape the DOM table and ignore the script tag, you'll get a name and maybe a phone number and nothing else. If you scrape the script tag and ignore the DOM, you'll get ratings and IDs but no address, no years-in-business, no website. You need both, merged, per business.

Why does the DOM only have half the data? 🕵️

When we built the BBB Business Leads Scraper, the first pass assumed BBB profile pages worked like most directory sites: a template with labeled fields you css_first() your way through. That assumption survives for exactly four fields — address, accreditation date, years in business, and website — which live nowhere except a <dt>/<dd> definition list further down the page.

Everything that actually matters for lead scoring — business name, BBB letter grade, accreditation status, phone number, the internal IDs BBB uses to build its own canonical URL — comes from a different place entirely: a webDigitalData object, wired into the page for BBB's own analytics vendor, that happens to be complete, well-formed JSON:

# actors/bbb-business-leads-scraper/src/parsers/common.py
WEB_DIGITAL_DATA_MARKER = "webDigitalData"

def extract_web_digital_data(html: str) -> dict | None:
    marker_pos = html.find(WEB_DIGITAL_DATA_MARKER)
    if marker_pos == -1:
        return None
    brace_start = html.find("{", marker_pos)
    raw = _extract_balanced_json(html, brace_start)
    ...
    return json.loads(raw)

That is not a regex grabbing {.*} between two markers — a naive non-greedy match snaps shut on the first stray } inside a nested object, which this blob has plenty of. It's a quote-aware, brace-depth scanner that walks the string character by character, tracking whether it's inside a JSON string (so a } inside quoted text doesn't fool it), and only returns once depth returns to zero. Cheap to write, easy to skip, and skipping it is exactly how a scraper ships a business_info blob that's silently truncated mid-object.

How do you match a JSON hit to its actual profile URL? 🔗

Search-results pages compound the problem. webDigitalData.search_info.results[] gives you one JSON object per business — name, BBB ID, rating, accreditation status — but no URL. The clickable profile link only exists in the static HTML anchor tags rendered alongside it. The two lists share order (each JSON hit's own position field confirms it: "page N, position M"), so we walk both lists by the same index and pair JSON hit i with DOM anchor i. A hit with no matching anchor, or an anchor that doesn't parse as a real profile URL, gets logged and skipped — not raised, not retried, just excluded, because the other 39 hits on that search page are still good data.

Every canonical BBB profile URL also encodes its own identity, which is the fallback path when a caller skips search entirely and hands the Actor direct profile URLs instead:

# https://www.bbb.org/{us|ca}/{region}/{city}/profile/{category}/{slug}-{bbb_id}-{business_id}
PROFILE_URL_PATTERN = re.compile(
    r"^https://www\.bbb\.org/(?:us|ca)/(?P<region>[a-z0-9]{2,3})/(?P<city>[a-z0-9-]+)/profile/"
    r"(?P<category>[a-z0-9-]+)/(?P<slug>[a-z0-9]+(?:-[a-z0-9]+)*?)-(?P<bbb_id>\d{3,6})-(?P<business_id>\d{6,})/?$"
)

Three sources of the same identity fields — JSON hit, DOM anchor, URL structure — with a defined priority order between them, is what makes both entry modes (category+location search, or a raw list of profile URLs) land in the same ResultRow shape without either one going stale when BBB tweaks a template.

What about the Cloudflare wall in front of all of it? 🛡️

None of the JSON parsing matters if the request never reaches BBB. bbb.org fronts every page — search and profile alike — with Cloudflare bot management, and we rotate through it the standard way: we rotate Chrome / Firefox / Safari TLS fingerprints via curl-cffi impersonation, we rotate residential-tier proxy sessions on every block, and we retry with exponential backoff (2s doubling to a 30s cap, five attempts) on 408/429/503. A 403 specifically triggers an immediate session rotation rather than a backoff-and-retry — recon confirmed 403 is this target's live Cloudflare-challenge status, so there's no point waiting it out on the same fingerprint.

One proxy group came back from cloud recon as the only one that actually clears bbb.org's wall in this account, and the Actor hardcodes it rather than exposing it as an overridable default — a caller who quietly reverts to a generic proxy group gets a silently-blocked run instead of a working one.

The part that generalises 🧭

BBB is not unusual in doing this. A lot of directory sites keep an internal analytics data layer that happens to be a more complete, more structural source of truth than the page's own visible markup — because the analytics vendor needs clean typed fields, and the template author doesn't. When a target's DOM feels thinner than the page obviously knows, check the <script> tags for a data-layer object before concluding the field just isn't there.

What the Actor gives you

• Business name, BBB letter-grade rating, accreditation status and date, phone, address, website, category, years in business, complaint count, review count, and average review score — one row per business.
• Two input modes in the same run: searchQueries (category + location discovery) and profileUrls (direct enrichment), deduped by business_id, first-seen wins.
• Per-business fault isolation — one unparseable or blocked profile is skipped and logged, never crashes the batch.
• Pydantic-validated rows with ISO-8601 timestamps.

Honest limitations 🚧

Full complaint narratives and review text aren't in scope — only the aggregate counts and scores BBB itself publishes. US/CA BBB directories only, no international equivalents, and it's a one-shot snapshot per run with no rating-history tracking.

FAQ

Do I need a BBB account or API key?
No. This scrapes bbb.org's own publicly listed business-profile data — no login, no key.

Can I mix category search and direct profile URLs in one run?
Yes. searchQueries and profileUrls both run in the same call, deduped by business_id.

Does this scrape review text or complaint narratives?
No — only aggregate counts and scores (review_count, complaint_count, average_review_score).

Why does my run sometimes come back slower than expected?
BBB's Cloudflare wall means every blocked request costs a fingerprint rotation and a backoff cycle before the next attempt — that's deliberate pacing, not a bug.

Pricing

$0.20 per run plus $0.0012 per business scraped — $1.40 per 1,000 leads. A run that matches nothing costs only the start fee.

→ BBB Business Leads Scraper on Apify

---

Built by Devil Scrapes. We handle the fingerprint rotation, the Cloudflare wall, and the JSON-vs-DOM merge, so you get a flat table instead of a weekend.