DEV Community

agenthustler
agenthustler

Posted on

Scraping Municipal Permit and Zoning Data for Real Estate Analysis

#python #tutorial #webdev #programming

Scraping Municipal Permit and Zoning Data for Real Estate Analysis

Building permits and zoning changes are leading indicators in real estate. When a city issues a wave of commercial permits, property values shift. This data is public but buried in municipal websites.

Why This Data Matters

  • Predict neighborhood development before it happens
  • Identify upcoming commercial zones
  • Track construction activity as an economic indicator
  • Find undervalued properties near planned developments

Setup 

import requests
from bs4 import BeautifulSoup
import pandas as pd
from datetime import datetime

PROXY_URL = "https://api.scraperapi.com"
API_KEY = "YOUR_SCRAPERAPI_KEY"
Enter fullscreen mode Exit fullscreen mode

Municipal sites vary wildly in structure. ScraperAPI with JavaScript rendering handles even legacy government portals.

Scraping Building Permits 

def scrape_permits(city_portal_url, pages=10):
    permits = []
    for page in range(1, pages + 1):
        params = {
            "api_key": API_KEY,
            "url": f"{city_portal_url}?page={page}",
            "render": "true"
        }
        response = requests.get(PROXY_URL, params=params)
        soup = BeautifulSoup(response.text, "html.parser")

        for row in soup.select("table.permits tbody tr"):
            cells = row.select("td")
            if len(cells) >= 5:
                permits.append({
                    "permit_number": cells[0].text.strip(),
                    "address": cells[1].text.strip(),
                    "type": cells[2].text.strip(),
                    "description": cells[3].text.strip(),
                    "status": cells[4].text.strip(),
                    "scraped_at": datetime.now().isoformat()
                })
        import time; time.sleep(3)
    return permits
Enter fullscreen mode Exit fullscreen mode

Analyzing Permit Trends 

def analyze_permit_trends(permits):
    df = pd.DataFrame(permits)
    type_counts = df["type"].value_counts().to_dict()
    commercial = df[df["type"].str.contains("commercial|business", case=False, na=False)]
    residential = df[df["type"].str.contains("residential|dwelling", case=False, na=False)]

    return {
        "total_permits": len(df),
        "by_type": type_counts,
        "commercial_count": len(commercial),
        "residential_count": len(residential),
        "commercial_ratio": round(len(commercial) / len(df) * 100, 1) if len(df) > 0 else 0
    }

def find_hotspots(permits, min_permits=5):
    df = pd.DataFrame(permits)
    import re
    streets = df["address"].str.extract(r'(\d+ .+? (?:St|Ave|Blvd|Rd|Dr))', expand=False)
    counts = streets.value_counts()
    return counts[counts >= min_permits].to_dict()
Enter fullscreen mode Exit fullscreen mode

Change Detection 

import sqlite3

def init_permit_db():
    conn = sqlite3.connect("permits.db")
    conn.execute('''CREATE TABLE IF NOT EXISTS permits (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        permit_number TEXT UNIQUE, address TEXT,
        type TEXT, description TEXT, status TEXT, scraped_at TEXT
    )''')
    conn.commit()
    return conn

def detect_new_permits(conn, permits):
    new_permits = []
    for p in permits:
        try:
            conn.execute(
                "INSERT INTO permits (permit_number, address, type, description, status, scraped_at) VALUES (?,?,?,?,?,?)",
                (p["permit_number"], p["address"], p["type"], p["description"], p["status"], p["scraped_at"])
            )
            new_permits.append(p)
        except sqlite3.IntegrityError:
            pass
    conn.commit()
    return new_permits
Enter fullscreen mode Exit fullscreen mode

Infrastructure

  • ScraperAPI — JavaScript rendering for legacy municipal portals
  • ThorData — residential IPs for geo-restricted local government data
  • ScrapeOps — monitor scraping health across multiple city portals

Conclusion

Permit and zoning data is the real estate market's crystal ball. Automate collection, track trends over time, and you'll spot development waves before they hit property prices.

Top comments (0)