DEV Community

Devil Scrapes
Devil Scrapes

Posted on

One Medicare API, Four Datasets, Three Renamed Fields

Quick answer

Medicare's Provider Data Catalog publishes facility data for nursing homes, hospitals, home health agencies and hospices through one keyless JSON API — and renames the same three fields in every single one of them. A client written against the nursing-home dataset does not return wrong data on the hospital dataset; it returns null, quietly, for the facility's own identifier. The CMS Provider Facilities Scraper maps all four datasets to one stable row — CCN, name, address, city, state, ZIP, phone, plus every quality-rating column the dataset carries — at $4.20 per 1,000 facilities.

The trap: one API, four naming conventions ðŸŠĪ

Every dataset in the catalog is reached the same way, datastore/query/{datasetId}/0, and each returns a results array of flat objects. That uniformity is the problem, because the columns inside are not uniform at all. Here is the same field across the four facility types, verbatim:

Field Nursing homes Hospitals Home health Hospices
Medicare ID cms_certification_number_ccn facility_id cms_certification_number_ccn cms_certification_number_ccn
Name provider_name facility_name provider_name facility_name
Street provider_address address address address_line_1 + address_line_2

The Medicare CCN — the federal identifier that makes this data joinable to anything else — is called cms_certification_number_ccn in three datasets and facility_id in the fourth. Hospices are the only type that splits the street across two columns, so a client that reads address alone loses the suite number on every hospice in the country.

None of this raises an error. results is present, the row count is right, the HTTP status is 200, and the fields you asked for simply are not there. You get a clean, successful, complete-looking export in which the identifier column is empty — which is worse than a failure, because a failure tells you.

This Actor keeps a per-dataset field map and normalises all four into one row shape, so ccn means the Medicare ID whichever facility type you asked for. The columns that genuinely differ — star ratings, staffing hours, inspection dates, ownership — are preserved in a typed extras mapping rather than dropped to make the schemas match, because those columns are usually the reason someone pulls this data at all.

Why the row count is free, and why that matters ðŸ’ļ

The response carries a count field, and count is the total number of rows matching your filter — not the number returned in this page. Combined with server-side filtering, that turns a potentially enormous pull into a cheap one.

Filtering is done with a bracket-indexed query syntax that is easy to get subtly wrong:

conditions[0][property]=state
conditions[0][value]=CA
conditions[0][operator]==
Enter fullscreen mode Exit fullscreen mode

Note the == at the end — that is the parameter name operator followed by the value =, the equality operator. Get the index wrong, or omit the operator, and the filter is ignored rather than rejected: you page through every facility in the United States believing you asked for California.

Because count is the true total, paging can stop exactly on it instead of probing for an empty page, and a state filter is applied by CMS rather than by us after the download. That is not just tidier — external data transfer is the single largest line item on a scraping bill, so filtering server-side is the difference between pulling one state and pulling the country to throw most of it away.

Underneath, the run still has to survive an ordinary day against a federal endpoint: 429s and 5xxs under load get capped exponential backoff that honours Retry-After, a malformed record is validated, logged and skipped rather than sinking the page it arrived in, and a filter that legitimately matches nothing finishes as a clean successful run instead of a false failure.

Is CMS provider data hard to scrape? ðŸ›Ąïļ

Not in the anti-bot sense, and we will not pretend otherwise — data.cms.gov is a public federal open-data API with no challenge page, and we probed it before writing a line of client code. The difficulty here is not getting a response. It is that four datasets which look interchangeable are not, and the way they disagree produces silent nulls rather than loud errors. That is the class of bug that ships to production and is found a quarter later by someone reconciling a join.

FAQ

Which facility types are covered?
Nursing homes, hospitals, home health agencies and hospices — selected by name, not by pasting a dataset ID.

Do I get the quality ratings?
Yes. Every dataset-specific column, including star ratings and staffing measures, is preserved alongside the normalised core fields.

Is this the same as an NPI lookup?
No. NPI registries return individual practitioners. This returns facilities with their Medicare certification numbers and Care Compare quality columns — a different join key and a different buyer.

How is it billed?
$0.20 per run start plus $0.004 per facility row, so 1,000 facilities cost $4.20. You pay for rows that land.


Built by Devil Scrapes. We publish the traps we hit, because the ones that return 200 OK are the expensive ones.

Top comments (0)