Building a 12-City US Building Permits Scraper With Python

#python #webscraping #opensource #tutorial

Building permits are the earliest public signal of construction activity — new housing, commercial builds, renovation waves. Every major US city publishes them, but each through a different portal with a different schema. I built one scraper that covers twelve of them with a single output shape.

The gap

The reference actor in this niche covers the big cities but has blind spots: in its own all-cities run it scanned zero rows for Washington DC and never even attempted West Sacramento. Ours covers all twelve: ten Socrata portals (Chicago, NYC, LA, San Francisco, Austin, Seattle, Cincinnati, Mesa, Montgomery County, Baton Rouge) plus two ArcGIS servers (DC, West Sacramento).

The reverse-engineering

Two very different API families to reconcile:

Socrata — the well-documented resource endpoint family, clean JSON
ArcGIS — the old FeatureServer contract, where field names vary per server and some servers reject modern ordering parameters outright

The resilience ladder for the old ArcGIS servers: try TIMESTAMP + orderBy first, fall back to where-only queries, then to unfiltered pulls with client-side date filtering. West Sacramento's server rejects TIMESTAMP entirely — the ladder is what makes it work.

One schema to rule them all

Eighteen canonical keys: city, permit_number, permit_type, status, issue_date, application_date, address, work_description, valuation, contractor_name, applicant, latitude, longitude, source, source_type, source_url, scraped_at, cityKey. NYC's "Permit is not yet issued" placeholder rows get filtered. Adding a city is one entry in the config table.

The smoke test

24/24 non-zero — two runs per city across all twelve, full 18-key schema on every item, plus local unit tests covering ten Socrata normalisations, both ArcGIS servers, and live fetches.

The honest bits

Coverage is US-only and city-scoped — counties and rural areas aren't in this pass
Valuation can be null on older records; some cities simply don't publish it
ArcGIS servers rate-limit; the ladder paces requests per city

Try it

👉 Building Permits Scraper on Apify Store

DEV Community