Building permits are the earliest public signal of construction activity — new housing, commercial builds, renovation waves. Every major US city publishes them, but each through a different portal with a different schema. I built one scraper that covers twelve of them with a single output shape.
The gap
The reference actor in this niche covers the big cities but has blind spots: in its own all-cities run it scanned zero rows for Washington DC and never even attempted West Sacramento. Ours covers all twelve: ten Socrata portals (Chicago, NYC, LA, San Francisco, Austin, Seattle, Cincinnati, Mesa, Montgomery County, Baton Rouge) plus two ArcGIS servers (DC, West Sacramento).
The reverse-engineering
Two very different API families to reconcile:
-
Socrata — the well-documented
resourceendpoint family, clean JSON - ArcGIS — the old FeatureServer contract, where field names vary per server and some servers reject modern ordering parameters outright
The resilience ladder for the old ArcGIS servers: try TIMESTAMP + orderBy first, fall back to where-only queries, then to unfiltered pulls with client-side date filtering. West Sacramento's server rejects TIMESTAMP entirely — the ladder is what makes it work.
One schema to rule them all
Eighteen canonical keys: city, permit_number, permit_type, status, issue_date, application_date, address, work_description, valuation, contractor_name, applicant, latitude, longitude, source, source_type, source_url, scraped_at, cityKey. NYC's "Permit is not yet issued" placeholder rows get filtered. Adding a city is one entry in the config table.
The smoke test
24/24 non-zero — two runs per city across all twelve, full 18-key schema on every item, plus local unit tests covering ten Socrata normalisations, both ArcGIS servers, and live fetches.
The honest bits
- Coverage is US-only and city-scoped — counties and rural areas aren't in this pass
- Valuation can be null on older records; some cities simply don't publish it
- ArcGIS servers rate-limit; the ladder paces requests per city
Try it
👉 Building Permits Scraper on Apify Store
More from me
While you're here, these might be worth a read:
- I Built a Canada Product Recalls & Safety Alerts Scraper That Reads Open Government Data
- I Built a Telegram Members Scraper That Reads Public Chat Stats Without Login
- Building a WHOIS & DNS Lookup Tool: Domain Intelligence in One API Call
- Building an AI Web Crawler That Outputs LLM-Ready Content Chunks
- Building a Real-Time Press Release Monitor with Python and RSS Aggregation
- Building a Universal Property Listing Scraper with Python and JSON-LD
- Tracking Tech Sentiment in Real-Time with VADER and Python
- How I Built a Product Hunt Scraper That Tracks Launches in Real-Time
- 5 APIs Every Developer Needs for Content Processing
- How to Extract Clean Content From Any Website Sitemap
- Scraping 187,000 Romanian Businesses: Building a B2B Lead Generation Tool
- Make Any Website AI-Readable: Generating llms.txt Files with Python
- I Built an RSS Aggregator That Extracts Full Article Content
Top comments (0)