Rewriting the Civic Scraper: From Monolith to Microservices

When a municipal server changes its cert bundle, a monolithic scraper dies quietly in the dark. For months, our civic data pipeline lived inside a single, sprawling Python script fetching zoning laws, public notices, and city council minutes all at once. If the council page threw a timeout, the whole run halted. We lost hours of historical data to unhandled socket errors and memory leaks.

Municipal websites run on brittle, outdated infrastructure with terrible uptime. Relying on a single script meant a broken CSS selector on the parks department page could blind us to emergency zoning changes. We needed isolation, predictability, and a way to test components without hammering live government servers.

We rewrote the architecture in Go, breaking the monolith into single-purpose microservices running in Docker. Go gives us strong static typing, fast compilation times, and tiny memory footprints when spinning up dozens of concurrent workers. Each service handles one job. One container fetches raw HTML from the city council portal. Another parses the DOM for ordinance numbers. A third validates the schema before hitting the database.

Containerization changed how we handle network failures. By isolating the scraping logic inside Docker, we inject mock HTTP responses during testing. We stopped hitting live municipal servers during CI runs. If parsing logic for planning commission agendas fails, only that specific container crashes and restarts via our orchestration policy. The rest of the civic data flow keeps running.

Here is a stripped-down example of how we structure the worker payload in Go, passing raw HTML bytes through a decoupled channel for parsing:

type PagePayload struct {
 SourceURL string
 Body []byte
 Timestamp time.Time
}

func ParseOrdinance(ch <-chan PagePayload) {
 for payload := range ch {
 // Isolation ensures a malformed page only panics this worker
 if len(payload.Body) == 0 {
 continue
 }
 processDocument(payload.SourceURL, payload.Body)
 }
}

This pattern forces us to think about failure domains. When a city updates its web template, the blast radius stays contained to a single microservice rather than taking down the data warehouse. We can deploy a fix for the zoning scraper without risking the council minutes parser.

Government transparency depends on our systems outlasting the bureaucracy they monitor. By ditching brittle monoliths for modular backend design, we build tools communities can actually trust.