300 cities, 300 column names: normalizing Socrata building permits

Quick answer

Every US city publishing building permits on a Socrata open-data portal is handing contractors a pre-CRM buying signal — someone just paid a government fee to build something, and they need HVAC, roofing, solar, or a dozen other trades next. The catch: over 300 cities run Socrata, and none of them agree on what to call the same field. Chicago's project cost column is reported_cost; another city calls the identical concept valuation, another estimated_cost, another declared_value. The US Building Permit Leads Scraper normalizes across all of them into one schema, and prices at $0.002 per permit row plus a $0.03 run-start charge — $2.03 for 1,000 leads.

Twenty-one canonical fields, five-to-six raw names each 🏗️

Socrata's SODA API is a real, structured JSON API — no HTML to parse, no anti-bot fight. The hard part is entirely semantic: every municipality that publishes permits chose its own column names when it stood up its open-data portal, years apart, with no shared schema. permit_id alone might arrive as permit_, permitnum, permit_no, permit_id, or permitnumber depending on the city. contractor_name might be contact_1_name in one portal and licensed_contractor in another. The Actor carries an ordered alias list for each of its 21 canonical output fields and tries each raw column name in turn, taking the first non-empty match — so valuation resolves correctly whether the source portal called it reported_cost, estimated_cost, declared_value, or total_value.

That alias-matching approach has a deliberate failure mode: anything a portal calls that isn't in any alias list doesn't get silently dropped. It lands in the row's extra object, verbatim, keyed by its original column name. A city with an unusual schema still yields a complete row — you just find its idiosyncratic fields in extra instead of a named column.

The street address gets assembled, not copied whole 🏠

Most Socrata permit portals don't publish one clean address field — they publish street number, direction, name, and suffix as separate columns, because that's how the source municipal database stores it internally. The Actor pulls those parts through their own alias lists (street_number, street_direction, street_name, street_suffix) and assembles a single readable address string from whichever parts a given city actually populated, rather than requiring every portal to already ship a combined field it may not have.

What we handle for you 🛡️

• We rotate through Apify Proxy on every request, absorbing transient 429s and routing failures across a multi-city fan-out.
• We retry with exponential backoff on 429 / 503 and network errors, up to 5 attempts per request, honoring Retry-After.
• We normalize 21 canonical fields across incompatible city schemas via ordered alias lists, and preserve every unmapped column in extra instead of dropping it.
• We fan out across multiple cities in one run — pass an array of {domain, datasetId} pairs and every source is queried, tagged by source_domain.
• You pay only for permit rows that land. No data, no charge, beyond the small run-start fee.

Full output schema 📦

Twenty-two fields per permit record (21 canonical fields plus extra):

Field	Type	Notes
source_domain	string	Socrata portal hostname
dataset_id	string	Socrata four-by-four dataset ID
permit_id	string | null	Resolved from up to 6 raw column-name aliases
permit_type	string | null	e.g. "PERMIT - NEW CONSTRUCTION"
status	string | null	e.g. "permit issued"
work_description	string | null	Free-text scope of work
issue_date	string | null	ISO-8601 or raw string
application_date	string | null	Application submission date
address	string | null	Assembled from street number/direction/name/suffix
street_number, street_name, city, state, postal_code	string | null	Individual address components
valuation	float | null	Project value, resolved from up to 5 raw aliases
fee_paid	float | null	Building fee paid
contractor_name	string | null	Resolved from up to 5 raw aliases
contractor_license	string | null	License number, where published
applicant_name	string | null	Applicant or owner name
latitude / longitude	float | null	WGS-84 coordinates
extra	object | null	Every raw column that didn't map to a canonical field

Who this is for

HVAC, roofing, and solar contractor lead-gen — filter by permit type and harvest applicant or contractor contact frames for outbound before a competitor does.

Contractor-lead resellers — run nightly across multiple cities, normalize to one schema, and deliver fresh permit leads via the Apify API.

Real-estate and development analysts — track permit valuation and volume by ZIP code or district as a leading construction-activity indicator.

CRM enrichment — match permit applicant names and addresses against existing contacts to surface warm leads.

Frequently asked questions

Why do the same field names differ so much between cities?

Every Socrata portal was built independently by its own municipal IT team, often years apart, with no shared schema standard. The Actor's alias-list normalizer resolves the common naming variants for 21 canonical fields; anything outside those aliases lands in extra rather than getting dropped.

What happens to fields the Actor doesn't recognize?

They're preserved verbatim in the extra object, keyed by their original raw column name. Nothing a source portal publishes is discarded.

Do I need a Socrata API key?

No — every supported portal serves permit data on the public SODA endpoint without authentication.

Can I scrape multiple cities in one run?

Yes. Add multiple {domain, datasetId} objects to the datasets input array; the Actor fans out and tags every row with source_domain so you can filter by city downstream.

What does 10,000 permit records cost?

$20.03 — 10,000 × $0.002, plus the $0.03 run-start charge.

Try it

Live on the Apify Store: US Building Permit Leads Scraper.

Point it at any Socrata building-permit portal — or several at once — and get back one normalized schema regardless of how the source city named its columns. Pay-per-event, no subscription.

---

Built by Devil Scrapes — we build scrapers for the targets that fight back.