DEV Community

Devil Scrapes
Devil Scrapes

Posted on

Don't scrape data.gov. Speak CKAN.

#api #python #opendata #webscraping

Quick answer

There are thousands of government open-data portals running CKAN — data.gov, data.gov.uk, open.canada.ca, most European national portals, and a long tail of city and provincial sites. They all speak the same Action API. So the useful thing to build is not "a data.gov scraper" but one Actor that speaks the protocol, pointed at whichever portal you care about.

That is what the CKAN Open Data Portal Scraper is. Give it a portal URL and a query; get flat rows.

The protocol, in one block 🔌 

GET https://<portal>/api/3/action/package_search?q=<query>&rows=<n>&start=<offset>
GET https://<portal>/api/3/action/package_show?id=<dataset-id>
GET https://<portal>/api/3/action/organization_list
Enter fullscreen mode Exit fullscreen mode

Every response is wrapped:

{"success": true, "result": { ... }}
Enter fullscreen mode Exit fullscreen mode

Unwrap result. And check success — which brings us to the one place the documentation lied to us.

package_show on an unknown id returns 404, not 200 🚧

Our spec said an unknown dataset id comes back HTTP 200 with success: false. Confirmed live against both data.gov.uk and open.canada.ca: it is HTTP 404, with success: false in the body.

That distinction decides whether a bad id is a row-level skip or a run-level failure. If you treat every non-200 as fatal, one stale id in a list of five hundred kills the whole run and the customer pays the start fee for nothing. If you treat a 404 on package_show as "that one dataset is gone, keep going," you deliver the other 499.

This is the single most common defect shape we find in our own fleet: a recoverable error crashing the entire run instead of skipping one item. Every time we fix it, success rates move.

The bug that would have billed 200x for a "small demo" 💸

Worth confessing, because it is subtle and it nearly shipped.

The QA fixture and the Store prefill both set rows: 5 — a tiny demo, obviously. Except rows is the CKAN page size, and the Actor's own max_items cap still sat at its default of 1000. So the "5-row demo" would have politely fetched five rows at a time, two hundred times over, per portal.

Nothing about that is a crash. It succeeds, it returns data, every test passes. It just costs two hundred requests to demonstrate five rows, on an Actor that bills per run.

Two knobs that sound like the same thing are not the same thing: page size is how you ask; item cap is how much you want. Both need setting, and the demo needs both set small.

Why a protocol harvester rather than a site scraper 🎯

We measure how crowded a niche is before building, using the Apify Store's own search API — how many listings already do this, and do any of them have paying users.

ckan came back at 7 rivals, 1 with users. That is the shallowest live cluster we have measured, against terms where 30 to 250 rivals is normal. The reason is structural: most people build "a data.gov scraper", solve one portal, and stop. The protocol is the thing that generalises, and almost nobody sells it.

Practically, that means one Actor and one output schema across:

  • package_search — full-text search with faceting and filter queries
  • package_show — complete metadata for a known dataset id
  • organization_list — the publishing bodies on a portal

Point it at a national portal, a city portal, or a research repository. Same rows.

What a row looks like

Portal URL, the action used, the query, the record id, an ISO-8601 scrape timestamp, and the dataset payload — name, title, notes, licence, organisation, resource count, created/modified timestamps, and the resource list.

Verified end-to-end against data.gov.uk and open.canada.ca, and spot-checked back against the portals themselves: the ids and names in our dataset match what package_show returns for the same records. Rows existing is not proof they are real — we check that they are.

Who this is for

  • Civic tech and journalism — find every dataset a government publishes on a topic, across countries.
  • Data engineering — discover and monitor open-data assets without writing a client per portal.
  • Research ops — licence and provenance metadata in a consistent shape.

The honest limitations 🚧

  • Metadata harvesting, not file downloading — you get the resource URLs, not the CSVs behind them.
  • Portals on very old CKAN versions may not expose every action.
  • Some portals rate-limit anonymous API access; we retry with backoff and respect it.

Pricing

$0.05 per run plus $0.003 per row — $3.05 per 1,000. Pay for rows that land.

→ CKAN Open Data Portal Scraper on Apify

Built by Devil Scrapes. We handle the envelopes, the pagination, the 404s that should not kill a run, and the portals that disagree with the docs.

Top comments (0)