Building a Reliable FMCSA Carrier Data Scraper with Python and Playwright

TL;DR: Public FMCSA carrier records are useful for logistics research, but a fragile script will get you blocked or produce dirty data. This guide walks through a production-minded Playwright scraper: respectful rate limits, retries with backoff, MC/USDOT hygiene, and clear compliance boundaries—without inventing shortcuts past access controls.

Why FMCSA data, and why Playwright?

U.S. freight and brokerage teams routinely look up carriers by MC or USDOT number: authority status, legal name, address, and equipment hints. Much of that is published as public records. The hard part is not “can I fetch HTML?”—it is keeping a long-running collector stable when pages change, when the network flakes, and when you need clean, deduplicated rows for a CRM or directory.

Playwright fits this job because it:

• Speaks modern Chromium (and Firefox/WebKit) with a stable API
• Handles waits for network and DOM more predictably than older Selenium patterns
• Gives you request interception, tracing, and storage state for sessions you are allowed to use

This article is about authorized, public-page collection. It is not about bypassing logins, CAPTCHAs, paywalls, or private APIs. If a site’s terms require written permission for automated access, get that first.

A minimal, honest architecture

A scraper that lasts usually has four layers:

1. Seed list — MC/USDOT numbers or search queries you already have a right to process
2. Fetcher — Playwright (or HTTP where a stable public API exists)
3. Normalizer — parse → typed fields → MC/USDOT cleanup → dedupe keys
4. Sink — CSV/SQLite/Postgres plus a run log (what succeeded, what failed, why)

Keep secrets out of the repo. Put rate limits and user-agent policy in config, not hard-coded in five places.

# config.py — illustrative defaults, tune per target and permission
RATE_LIMIT_SECONDS = 2.5      # pause between page navigations
MAX_RETRIES = 4
BACKOFF_BASE = 2.0            # exponential: 2, 4, 8, 16...
REQUEST_TIMEOUT_MS = 45_000
USER_AGENT = (
    "INDUSResearchBot/1.0 (+https://www.induswebagency.com; research; contact=[e-mail])"
)

Identify your bot. A clear User-Agent and a contact URL make it easier for operators to reach you if something goes wrong—and it is basic good citizenship.

Fetching with Playwright: waits, not sleeps everywhere

Prefer Playwright’s auto-waiting locators over fixed time.sleep calls. Sleeps still belong in the rate limiter between independent navigations.

from playwright.sync_api import sync_playwright, TimeoutError as PlaywrightTimeout
import time, random

def polite_pause(base: float = RATE_LIMIT_SECONDS) -> None:
    # small jitter so traffic is not perfectly periodic
    time.sleep(base + random.uniform(0.1, 0.6))

def fetch_carrier_page(page, url: str) -> str:
    page.goto(url, wait_until="domcontentloaded", timeout=REQUEST_TIMEOUT_MS)
    # Wait for a stable landmark on the public record page—adjust selector to the live DOM
    page.wait_for_selector("[data-carrier-record], .carrier-details, main", timeout=20_000)
    return page.content()

Use domcontentloaded when you do not need every third-party asset. Prefer a specific selector for the record block once you know the page structure.

Retries that survive flaky networks

Retries should be idempotent and bounded. Retry timeouts and 5xx-class failures; do not blindly retry on “record not found.”

from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type

class TransientFetchError(Exception):
    pass

@retry(
    reraise=True,
    stop=stop_after_attempt(MAX_RETRIES),
    wait=wait_exponential(multiplier=1, min=2, max=60),
    retry=retry_if_exception_type((PlaywrightTimeout, TransientFetchError)),
)
def fetch_with_retries(page, url: str) -> str:
    try:
        html = fetch_carrier_page(page, url)
        # Example: detect a soft error banner if the site returns 200 with an error body
        if "temporarily unavailable" in html.lower():
            raise TransientFetchError("soft 503-style body")
        return html
    except PlaywrightTimeout:
        raise

Log every attempt: URL, attempt number, exception type, elapsed ms. Without that log, “the scraper is broken” becomes guesswork.

Data hygiene: MC, USDOT, and dedupe keys

Dirty logistics data is worse than no data. Normalize before you store.

import re

MC_RE = re.compile(r"^(?:MC)?[\s\-]?(\d{1,8})$", re.I)
DOT_RE = re.compile(r"^(?:USDOT|DOT)?[\s\-]?(\d{1,9})$", re.I)

def normalize_mc(raw: str | None) -> str | None:
    if not raw:
        return None
    m = MC_RE.match(raw.strip())
    return f"MC{int(m.group(1))}" if m else None  # strip leading zeros via int()

def normalize_usdot(raw: str | None) -> str | None:
    if not raw:
        return None
    m = DOT_RE.match(raw.strip())
    return f"USDOT{int(m.group(1))}" if m else None

def dedupe_key(mc: str | None, usdot: str | None, legal_name: str | None) -> str:
    if usdot:
        return usdot
    if mc:
        return mc
    # last resort — never preferred
    return (legal_name or "").strip().upper()

Also:

• Trim and collapse whitespace in names and addresses
• Uppercase state codes; keep ZIP as text so leading zeros survive
• Store source_url and collected_at (UTC) on every row
• Reject rows with neither MC nor USDOT unless your use case truly needs name-only stubs

Staying compliant (non-negotiable)

Compliance is part of the engineering, not a footer disclaimer.

• Public pages only unless you have written authorization for authenticated areas
• No CAPTCHA/paywall bypass, no credential stuffing, no private API reverse engineering presented as “scraping tips”
• Respect robots.txt and terms for the specific host you hit; when terms require permission, ask
• Rate limit so you do not look like a denial-of-service
• PII: do not scrape personal emails/phones from places that are not meant for bulk collection; carrier company public records are a different category from personal contact harvesting
• Purpose limitation: keep the collector scoped to research / CRM enrichment you are allowed to do

If you ship a tool for others (browser extension or CLI), put explicit authorization checkboxes in the UI before any multi-page crawl. That pattern shows up in responsible logistics tooling for a reason.

Putting it together: a small run loop

def run(seeds: list[dict]) -> None:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        context = browser.new_context(user_agent=USER_AGENT)
        page = context.new_page()
        for seed in seeds:
            url = seed["url"]
            polite_pause()
            try:
                html = fetch_with_retries(page, url)
                record = parse_carrier_html(html)  # your parser
                record["mc"] = normalize_mc(record.get("mc"))
                record["usdot"] = normalize_usdot(record.get("usdot"))
                record["dedupe_key"] = dedupe_key(
                    record["mc"], record["usdot"], record.get("legal_name")
                )
                record["source_url"] = url
                upsert(record)  # SQLite/Postgres with unique(dedupe_key)
            except Exception as exc:
                log_failure(url, exc)
        browser.close()

Start with a dry-run mode that writes parsed JSON to disk and never upserts. Validate 20–50 known carriers by hand before you trust a full seed list.

Operational checklist

• Headless CI job with a nightly seed slice, plus a manual “full refresh” path
• Schema migrations when the DOM changes (version your parser)
• Alerts when failure rate crosses a threshold (example: >5% hard failures in a run)
• Export that matches what humans expect: CSV with stable column names for ops teams

FAQ

Q: Can I scrape FMCSA-related sites as fast as my connection allows?

A: No. Bound concurrency, pause between navigations, and follow the host’s terms. Speed is not a feature if it gets you blocked or creates legal risk.

Q: Should I use raw HTTP instead of Playwright?

A: Use HTTP when the data is available from a documented public endpoint or a simple static page. Use Playwright when the record is rendered by JavaScript or needs a real browser context you are allowed to drive.

Q: How do I avoid duplicate carriers across runs?

A: Normalize MC/USDOT first, then upsert on USDOT (preferred) or MC. Keep a run log so you can re-fetch only failures.

Q: Is storing public carrier company data the same as scraping personal contacts?

A: No. Company authority records and personal phone/email harvesting are different risk profiles. Stay away from personal contact scraping unless you have a clear lawful basis and permission.

---

Need a production-minded Python scraper or carrier directory for logistics ops? INDUS Web Agency builds compliant automation for freight and dispatch teams.