Query every dentist (or physician) in a metro without hitting NPPES's 1,200-record wall

Every US healthcare provider — every physician, dentist, nurse practitioner, physical therapist, pharmacy, and clinic — has a National Provider Identifier, and the whole registry is public. CMS runs it as NPPES, and it has a free, keyless API. If you sell to providers (dental supplies, medical devices, EHR/billing software, staffing), this is the authoritative list of your entire market, with practice addresses and phone numbers, for free.

There's one catch, and it's the reason so many "NPI scrapers" quietly return garbage: the NPPES API hard-caps every search at 1,200 records — and past that cap it doesn't error, it silently repeats the same page. If a metro has 4,000 dentists and you page naively, you'll get 1,200 real ones followed by duplicates of the last page, feel like you got "a lot of data," and never realize two-thirds of your market is missing.

This post explains the wall, shows the manual way through it, and gives you a copy-paste Python fix.

The API, and the wall

The base call is simple. Dentists in Florida:

curl "https://npiregistry.cms.hhs.gov/api/?version=2.1&\
taxonomy_description=Dentist&state=FL&limit=200"

limit maxes out at 200 per call. To go deeper you add skip:

curl "https://npiregistry.cms.hhs.gov/api/?version=2.1&\
taxonomy_description=Dentist&state=FL&limit=200&skip=1000"

And here's the wall: skip maxes out at 1,000. 1,000 skipped + 200 returned = 1,200 records, full stop. Ask for skip=1200 and the API doesn't say "out of range" — it hands you the skip=1000 page again. Naive pagers treat that as more data and append duplicates forever. (You'll also notice very broad searches — a whole state with no other filter — get rejected outright; NPPES requires at least one criterion beyond state.)

So the real problem isn't fetching data. It's fetching complete data for any search that legitimately has more than 1,200 matches — which is most useful ones (dentists in Miami, PTs in Houston, family medicine in all of California).

The manual fix: subdivide until every slice fits under 1,200

The trick is to split one too-big search into several small-enough searches, then merge and de-duplicate. The cleanest axis to split on is ZIP code prefix, because NPPES supports trailing wildcards on postal_code. "All Miami dentists" (too big) becomes "dentists in 331xx," then if that's still over 1,200, "3310x, 3311x, 3312x…," and so on. You recurse only into the slices that are actually full, and you de-duplicate the final set by NPI (a provider with a practice and a mailing address in different ZIPs can match twice).

Here's the whole thing in Python — no dependencies beyond requests:

import requests

API = "https://npiregistry.cms.hhs.gov/api/"
CAP = 1200          # NPPES hard cap per search
PAGE = 200          # max limit per call

def _page_all(params):
    """Fetch up to the 1,200 cap for one exact search."""
    out, skip = [], 0
    while skip <= 1000:
        r = requests.get(API, params={**params, "version": "2.1",
                                       "limit": PAGE, "skip": skip}, timeout=30)
        r.raise_for_status()
        results = r.json().get("results", [])
        if not results:
            break
        out.extend(results)
        if len(results) < PAGE:
            break
        skip += PAGE
    return out

def search(base, zip_prefix=""):
    """Recursively subdivide by ZIP prefix so no slice exceeds the cap."""
    params = dict(base)
    if zip_prefix:
        params["postal_code"] = zip_prefix + "*"
    hits = _page_all(params)
    # If we came back pinned at the cap, this slice is probably truncated —
    # split it into 10 narrower ZIP prefixes and recurse.
    if len(hits) >= CAP and len(zip_prefix) < 5:
        deeper = []
        for d in "0123456789":
            deeper.extend(search(base, zip_prefix + d))
        return deeper
    return hits

def dedupe(records):
    seen, out = set(), []
    for rec in records:
        npi = rec.get("number")
        if npi and npi not in seen:
            seen.add(npi)
            out.append(rec)
    return out

# Every dentist in greater Miami, complete:
raw = search({"taxonomy_description": "Dentist", "state": "FL"}, zip_prefix="331")
providers = dedupe(raw)
print(f"{len(providers)} unique dentists (naive paging would have stopped at 1,200)")

Run that against a dense metro and you'll see it sail past 1,200 — a Miami-area dentist search returns close to 2,000 unique providers, not the 1,200 a single search reports. That gap is exactly the providers a naive scraper never sees.

A few things to add before you rely on it:

• Be polite. Add a small delay between calls and a retry/backoff — NPPES will throttle a tight loop. The recursion above can fire dozens of calls for a dense metro.
• Pull the fields you actually need out of each record: number (the NPI), the taxonomy marked primary for specialty and license, and the address where address_purpose == "LOCATION" for the practice phone. NPPES nests these in arrays, so a naive record["phone"] won't exist.
• Emails aren't in here. NPPES publishes practice phone, fax, and address, plus the authorized official's name and phone for organizations — but not provider email. Any tool selling you "NPI emails" is enriching from somewhere else; treat those with the skepticism they deserve.

The maintained shortcut

I wrapped this exact approach — the cap detection, the ZIP fan-out, the de-duplication, the field extraction, the backoff — into an Apify actor so you don't have to own the recursion:

NPI Registry Scraper (NPPES Healthcare Providers)

The complete-Miami-dentists query becomes:

{
    "taxonomyDescription": "Dentist",
    "state": "FL",
    "postalCode": "331*",
    "maxResults": 2000
}

It returns clean, flat records — npi, name, credential, specialty, licenseNumber, licenseState, practiceAddress (with phone and fax), authorizedOfficial for organizations — and hands you JSON/CSV/Excel, scheduling, and webhooks. You're charged per record returned, so a ~2,000-provider metro list costs about four dollars. The point of paying isn't access to the data (it's free and keyless); it's that the 1,200-cap fan-out, the paging, and the field-unnesting are already correct and maintained.

Which route should you take?

• One search that's clearly under 1,200 (a single small city and specialty) → just page it yourself with the first snippet. No fan-out needed.
• Any dense metro or statewide specialty, on a schedule, or feeding a CRM or AI agent → use the fan-out. The actor is the least-effort version of it (and it's callable as an MCP tool via Apify, so "how many pediatric dentists are in the Dallas metro?" becomes a one-line agent query).
• A one-off count for a single ZIP → honestly, the NPPES website is fine; don't automate a thing you'll do once.

The registry is public infrastructure paid for with tax dollars. The only thing standing between you and a complete provider list is knowing that 1,200 isn't the end — it's where the naive scrapers give up.

Hit a specialty or metro where the fan-out behaves oddly? Drop a comment — the taxonomy descriptions have some sharp edges (there are three different flavors of "nurse practitioner" alone) and I'm happy to compare notes.