How to Pull Building Permits from Any US City's Open-Data API (Socrata, ArcGIS, CKAN) in Python

#python #opendata #tutorial #api

If you've tried to get building permit data for more than one city, you've hit the same wall: there's no national permit API. Every city publishes its own feed, and almost all of them run on one of three platforms: Socrata, ArcGIS or CKAN. Learn those three query styles and you can read most US permit portals.

This guide gives you a working Python function for each platform, the paging and date quirks that break naive scripts, and a way to merge cities into one table. Every snippet below was tested on Oct 22, 2026 against Chicago (Socrata), Louisville (ArcGIS) and Pittsburgh (CKAN). Together they returned 5,165 permits issued in the last 30 days, 756 of them worth $100k or more.

How to tell which platform a city uses

You see in the URL	Platform	API shape
`/resource/xxxx-xxxx` or `data.<city>.gov/d/xxxx-xxxx`	Socrata	`https://<domain>/resource/<id>.json` + SoQL
`/arcgis/rest/services/.../FeatureServer/0` or a Hub site	ArcGIS	`<layer>/query?where=...&f=json`
`/dataset/<name>` with a "Data API" button	CKAN	`/api/3/action/datastore_search_sql`

To find a city's feed, search catalog.data.gov, hub.arcgis.com, or " open data building permits."

The three fetchers

import requests
from datetime import date, datetime, timedelta, timezone
SINCE = date.today() - timedelta(days=30)

def socrata(domain, dataset, date_field, limit=1000):
    url = f"https://{domain}/resource/{dataset}.json"
    out, offset = [], 0
    while True:
        page = requests.get(url, params={
            "$where": f"{date_field} >= '{SINCE}T00:00:00'",
            "$order": f"{date_field} DESC, :id",   # stable order for paging
            "$limit": limit, "$offset": offset}, timeout=120).json()
        out += page
        if len(page) < limit: return out
        offset += limit

def arcgis(layer, date_field, page_size=1000):
    out, offset = [], 0
    while True:
        r = requests.get(layer + "/query", params={
            "where": f"{date_field} >= DATE '{SINCE}'", "outFields": "*", "f": "json",
            "orderByFields": f"{date_field} DESC", "resultOffset": offset,
            "resultRecordCount": page_size}, timeout=120).json()
        feats = [f["attributes"] for f in r.get("features", [])]
        for a in feats:  # ArcGIS dates are epoch milliseconds
            if isinstance(a.get(date_field), (int, float)):
                a[date_field] = datetime.fromtimestamp(a[date_field] / 1000, timezone.utc).date().isoformat()
        out += feats
        if not r.get("exceededTransferLimit"): return out
        offset += len(feats)

def ckan(domain, resource_id, date_field, limit=5000):
    sql = (f'SELECT * FROM "{resource_id}" WHERE "{date_field}" >= \'{SINCE}\' '
           f'ORDER BY "{date_field}" DESC LIMIT {limit}')
    r = requests.get(f"https://{domain}/api/3/action/datastore_search_sql", params={"sql": sql}, timeout=120).json()
    return r["result"]["records"]

The gotchas these functions handle

Socrata paging needs a stable order. If you page with $offset but sort only by date, rows that share a timestamp can repeat or go missing between pages. Adding :id as a tie-breaker fixes it.
Socrata leaves null fields out of the JSON. Rows come back with different keys. Use .get() everywhere, or call the .csv endpoint if you need fixed columns.
ArcGIS dates are epoch milliseconds, and the where clause needs DATE 'YYYY-MM-DD' syntax. A plain string comparison silently returns nothing on many servers.
ArcGIS caps each page (often at 1,000 or 2,000 rows) and tells you there's more with exceededTransferLimit. Loop until it's false.
CKAN column names are case-sensitive and often contain spaces (San Antonio uses "DATE ISSUED"), so always double-quote them in SQL.
Text-typed numbers. Many portals store valuation as text. Cast before you sum: float(x or 0) in Python, or ::number in SoQL.

Use it: three cities, one table

chicago = socrata("data.cityofchicago.org", "ydr8-5enu", "issue_date")
louisville = arcgis("https://services1.arcgis.com/79kfd2K6fskCAkyg/arcgis/rest/services/"
                    "active_construction_permits/FeatureServer/0", "ISSUE_DATE")
pittsburgh = ckan("data.wprdc.org", "f4d1177a-f597-4c32-8cbf-7885f56253f6", "issue_date")

def norm(city, r, num, date_f, desc, value, addr, contractor=None):
    return {"city": city, "permit": r.get(num), "issued": str(r.get(date_f))[:10],
            "description": r.get(desc), "value": float(r.get(value) or 0),
            "address": r.get(addr), "contractor": (r.get(contractor) or "").strip() or None}

rows = ([norm("Chicago", r, "permit_", "issue_date", "work_description", "reported_cost", "street_name") for r in chicago]
      + [norm("Louisville", r, "PERMIT_NUMBER", "ISSUE_DATE", "WORK_TYPE", "PROJECT_COSTS", "ADDRESS", "CONTRACTOR") for r in louisville]
      + [norm("Pittsburgh", r, "permit_id", "issue_date", "work_description", "total_project_value", "address", "contractor_name") for r in pittsburgh])
rows.sort(key=lambda r: r["issued"], reverse=True)
big = [r for r in rows if r["value"] >= 100_000]
print(len(rows), "permits,", len(big), "over $100k")

From here it's normal pandas: group by ZIP, filter descriptions for solar|roof|hvac|pool|adu, or write it all to a CSV.

Making it work for more cities

Keep a config, not code. Store each city as (platform, endpoint, date field, field map) in a list. Adding a city then takes one line.
Normalize work types yourself. Each city words them differently ("RES-ELC", "Electrical", "E-Permit"), so map them to your own categories with keyword rules.
Watch for stale feeds. Some portals quietly stop updating. Log the newest issued date per city on every run and alert when it's more than about 14 days old.
Respect the portals. Use their documented APIs, page politely, and cache. Socrata gives higher rate limits if you register a free app token.

Optional, and the code above is all you need: I maintain a hosted version of this pattern covering 47 cities and counties in one schema, Building Permits Scraper on Apify (paid per result). Disclosure: it's my tool.

DEV Community