SuburbStory publishes local updates for every suburb in NSW. The data behind each update comes from several government and public feeds: police, fire, traffic, transport, and various agency alerts. Each of these sources is independent, updated on its own schedule, and exposed through completely different HTTP endpoints.
Because of that variety, the core engineering problem isn’t processing the data. It’s getting it fast enough, from all sources, without creating a slow and fragile chain of sequential API calls. The entire workload is I/O: pulling remote webpages, parsing lightweight text, and sending the processed information to Gemini to generate suburb-level summaries.
The only way to make this practical at state-wide scale is to run everything in parallel. That’s where Python’s asyncio becomes the centre of the system.
The Problem With Sequential Fetching
If you fetch each source one at a time, you create an artificial bottleneck. A single slow feed (for example a public incident feed that occasionally takes a few seconds to respond) blocks all other data from being processed. When you multiply that across hundreds of suburbs, you lose freshness and your cron job runtime becomes unpredictable.
But none of these tasks depend on each other. Each fetch is just an isolated HTTP call. There’s no reason to wait for one to finish before starting the next.
Using asyncio.gather to Hit Every Source at Once
The actual workflow is simple: as soon as the cron job starts, every data source is queried in parallel. Instead of a loop like:
for source in sources:
data = fetch(source)
process(data)
the pipeline builds a list of coroutines and launches them simultaneously:
results = await asyncio.gather(*tasks, return_exceptions=True)
Every request goes out at the same time. Each one resolves whenever the remote server responds. A slow feed no longer slows down anything else. A fast feed is processed immediately. When you have tens of government sources multiplied across thousands of suburbs, this difference is enormous.
Parsing and Grouping on the Fly
Once the fetches complete, the content is parsed and normalised. These steps are lightweight — mostly HTML extraction, field cleaning, text slicing, and location matching — so they run inline without blocking the loop. By the time all tasks in the gather call have resolved, you already have a full snapshot of every government update that matters for that cycle.
The next step is grouping by suburb. Because all sources arrive together, the system can assemble a complete picture of what happened in a suburb across police, fire, traffic, and other feeds in a single pass. That grouped data becomes the input to the NLG stage.
Parallel Model Requests to Gemini
Model calls are also I/O. They involve sending the suburb-level dataset to Gemini and waiting for a generated summary. Running these sequentially would be even slower than sequential scraping. But they fit the same pattern as the fetches: independent, high-latency network operations.
So the pipeline builds another batch of tasks (one model call per suburb) and fires them all at once using asyncio.gather, with concurrency limits where needed. While the model is working on one suburb, dozens of other tasks are in flight. No time is spent waiting for NLG work to complete before starting the next one.
This effectively compresses what would be minutes of serialized model calls down to the latency of a single batch.
The entire cycle looks like this:
- All data sources fetched concurrently
- All parsing done immediately after responses arrive
- All suburb summaries generated concurrently
- All output stored without blocking the loop
The cron job doesn’t wait on anything except the slowest item in each batch. Everything else is overlapped. Even though the system touches many external endpoints and calls an LLM at scale, the total runtime stays short and predictable because every part of the pipeline runs in parallel.
Why Asyncio Suits This Problem Exactly
The workload behind SuburbStory isn’t CPU-heavy. There’s no local ML model, no heavy processing, no long transformations. It’s almost entirely network-bound. Asyncio is designed for this exact shape of problem: many independent I/O operations that can safely run together.
Using threads or processes would add overhead and resource waste. Using synchronous requests would stretch each cron cycle unnecessarily. Asyncio sits neatly in the middle with minimal overhead, maximum concurrency, and predictable performance.
Top comments (0)