DEV Community

Cover image for How I Build 800+ Pages of Local Open Data with Jekyll, Python and a Raspberry Pi
thechelsuk
thechelsuk

Posted on

How I Build 800+ Pages of Local Open Data with Jekyll, Python and a Raspberry Pi

Cheltenham Open Data is a free site I run for my home town. It takes public data that's spread across council PDFs, government spreadsheets and APIs and puts it on plain, fast pages: fuel prices, planning applications, flood warnings, bus disruptions, care home ratings and a lot more.

The sitemap now has a little over 800 URLs. I didn't write 800 pages. About 225 are hand-written Markdown, around 100 are news and analysis posts, and the rest, over 500, are generated at build time from data. There's no database, no server and no framework beyond Jekyll. This post goes through how it works, and the parts I'd do the same way again.

I've written before about why I built it and why tools beat blog posts for a local site. This one is just about the technology.

The Shape of It

Every data page follows the same pipeline:

source API / CSV / PDF
        │
        ▼
_python/<name>.py      fetch, clean, filter to Cheltenham
        │
        ▼
_data/<name>.json      committed to git
        │
        ▼
_layouts/<group>/<name>.html   Liquid turns it into tables, maps, charts
        │
        ▼
_pages/<group>/<name>.md       front matter + the words a person reads
        │
        ▼
GitHub Pages
Enter fullscreen mode Exit fullscreen mode

That's it. Git is the database. Each fetcher writes a JSON file, a push triggers a build, and the build is the deploy.

There are about 90 Python fetchers and nearly 150 JSON data files. Sources include ONS and Nomis, HM Land Registry, the Bus Open Data Service, National Rail, National Highways, the Environment Agency, Ofcom, the Department for Education, the CQC, the Food Standards Agency, the GOV.UK fuel price scheme and OpenStreetMap.

Fetchers: Boring on Purpose

Each fetcher is a standalone script that writes one JSON file with the same envelope: the data plus generated_at, source, source_url and licence. That envelope means every page can print an honest "About This Data" line at the bottom without anyone having to remember to write one.

The most useful helper in the whole repo is ten lines long. write_json refuses to write a file if the only thing that changed is the run time stamp:

def write_json(path, payload):
    """Write payload as JSON, unless the file already holds the same data and
    only the run time stamps would change. Leaving the file alone means a run
    that finds nothing new makes no commit."""
    path = pathlib.Path(path)
    if path.exists():
        try:
            old = json.loads(path.read_text())
        except ValueError:
            old = None
        if _without_stamps(old) == _without_stamps(payload):
            return False
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(json.dumps(payload, indent=2, ensure_ascii=False))
    return True
Enter fullscreen mode Exit fullscreen mode

That one decision pays off three times:

  1. A fetcher that runs every five minutes and finds nothing new makes no commit, so the git history stays readable.
  2. The "Last updated" date on each page can come straight from git (more on that below), and it's the day the data actually changed, not the day a cron job last ran.
  3. No pointless rebuilds.

Every fetcher also runs through a small wrapper that records whether it worked in a status file, and a sources.json registry lists every source, its licence, its data files and how often each one should change. A pytest test fails if a data file isn't in the registry, so nothing gets added quietly without a licence attached.

Generating Hundreds of Pages Without Writing Them

Jekyll has a little-used feature that does most of the heavy lifting here: generator plugins. A plugin in _plugins/ can read site.data and add pages to the site before rendering, with no front-matter file on disk.

The site has three of these:

  • Care homes: the monthly CQC fetcher gives each care home and home care service a slug, and a plugin makes a page for each one. A newly registered home gets its page on the next build.
  • Schools: the same pattern for Ofsted inspection history, one page per school.
  • Cheltenham Town FC: a league history fetcher feeds pages for every season since 2000-01, every opponent and every referee. That alone is around 380 pages.

The pattern is roughly:

module CareHomePages
  class Generator < Jekyll::Generator
    def generate(site)
      site.data.dig("cqc", "locations").to_a.each do |home|
        site.pages << CareHomePage.new(site, home)
      end
    end
  end
end
Enter fullscreen mode Exit fullscreen mode

The plugins also handle the bits that are easy to get wrong at scale. Titles and meta descriptions are built from a list of options and the longest one that fits the 60 or 160 character limit wins, so a short name gets a rich title and a long one still fits in search results.

Because the site builds with bundle exec jekyll build in GitHub Actions rather than the old locked-down GitHub Pages builder, custom plugins are fine.

Plugins That Keep Pages Honest

A few smaller plugins run after rendering and stop the generated HTML drifting from what's on screen:

  • FAQ schema: every page writes its FAQs the same way, a ## Frequently Asked Questions heading, then ### questions with bullet answers. A plugin reads the rendered HTML, finds those pairs and writes the matching FAQPage JSON-LD into the <head>. The structured data can't disagree with the page because it's made from the page.
  • Data freshness: one git log pass dates every _data file by its last commit. Each page shows "Last updated …" from that, and the sitemap's <lastmod> is the later of the page's own last change and that of any data it shows. The deploy checks out full history (fetch-depth: 0 with filter: blob:none to keep it fast), because on a shallow clone every file would get the same date.
  • Table tidying: Kramdown's Markdown tables come out bare, so a plugin wraps them for sideways scrolling on phones, turns right-aligned columns into a number class, and stops postcodes and dates from wrapping mid-value.
  • Navigation: the menu, category pages, breadcrumbs, "In this section" rows and the A-Z footer all come from one YAML file per category. Adding a page to the nav means adding a few lines of YAML, and everything else follows.

Maps and the Front End

Maps use Leaflet 1.9.4 with OpenStreetMap tiles, each page with a small script of its own, about 80 of them in all, following the same IIFE shape. The crime map shades small areas by crimes per 1,000 residents, and the school catchment maps compute Voronoi regions in the browser.

There's no framework and no build step for the JavaScript. Sass is compiled by Jekyll. There are no tracking cookies and no ad tech, just anonymised, cookie-free page counts.

The same data also feeds free, self-updating data widgets that anyone can drop into their own site. They're just more Jekyll pages, built from the same JSON on every deploy.

Scheduling: Why a Raspberry Pi Runs the Show

This is the part I'd tell anyone starting something similar.

GitHub Actions' schedule: trigger is best effort. In practice my "hourly" job was running once or twice a day, and five-minute jobs were nowhere near five minutes, with nothing in the UI to say so. For a site where fuel prices and power cuts should be current, that's no good.

So no workflow in the repo has a schedule: any more. A Raspberry Pi at home does two jobs instead:

  1. It runs the fast feeds itself. Bus disruptions, power cuts, train times, unplanned road closures, weather warnings and fuel prices run from the Pi every five minutes or hourly. Planning applications run there too, because the council's planning portal blocks GitHub's IP ranges.
  2. It starts every other workflow on time, using workflow_dispatch from its crontab.
*/5 * * * *  flock -n /tmp/chelt-od.lock $PI/run.sh every-5-minutes
3 * * * *    flock -w 300 /tmp/chelt-od.lock $PI/run.sh hourly
36 */2 * * * $PI/pi/dispatch.sh refresh-every-2-hours.yml
23 4 * * MON $PI/pi/dispatch.sh refresh-weekly.yml
17 0 2 * *   $PI/pi/dispatch.sh refresh-monthly.yml
Enter fullscreen mode Exit fullscreen mode

Each run pulls main, runs the scripts in a manifest file, commits anything that changed and pushes, which triggers the Pages deploy. flock stops two runs pulling or pushing at the same time. The manifests are plain text files read after the pull, so adding a script to the Pi is just a commit. There's no SSH-ing in.

One small trick: run.sh puts its whole body in a function called on the last line. Bash reads scripts as it runs them, so without that, a git pull that changed run.sh partway through could run half the old file and half the new one.

A Cloudflare Worker as the Watchdog

A Pi on home Wi-Fi will go quiet sometimes. After each good run it posts a heartbeat to a Cloudflare Worker. Every five minutes the Worker checks the last check-in, and after two missed runs it takes over:

  • it starts a fallback workflow that runs the five-minute feeds in Actions
  • it starts any scheduled workflow that fell due while the Pi was down, unless GitHub shows it already ran
  • it emails me and turns a Hue light in the house orange, checking GitHub's status page first so a GitHub outage isn't blamed on the Pi

The next heartbeat hands control back. The Worker keeps its own copy of the schedule, and a test fails if it doesn't match the Pi's crontab, so the two can't drift apart.

Tests as Guard Rails

There are over 70 pytest files. Most are ordinary fetcher tests, but the ones I value most check the structure of the project:

  • every data file is in the sources registry, with a licence
  • every Pi script is in exactly one manifest
  • no workflow has a GitHub schedule:
  • the watchdog's schedule matches the crontab

None of these are clever. They exist because each one was a mistake I made once.

CI also runs super-linter, CodeQL, dependency review and a weekly link checker, which matters on a site with thousands of outbound links to official sources.

Getting Found

With 800 pages, search engines need help knowing what changed. After each deploy, a workflow compares sitemap <lastmod> dates against the commit of the last successful deploy and sends only the new or changed URLs to IndexNow. Because <lastmod> comes from real data changes, that list stays short and accurate.

The repository is public on GitHub and archived on Zenodo with a DOI, so researchers can cite it, and some have.

What It Costs

  • Hosting: GitHub Pages, free
  • Builds and most fetchers: GitHub Actions, free for a public repo
  • Watchdog: Cloudflare Workers free tier
  • Scheduling and fast feeds: a Raspberry Pi's electricity - most of this is solar-powered
  • the domain name for less than $5 a year.

The real cost is time. To cover that, the plan is a small number of local businesses each sponsoring one section, with a single "supported by" card and nothing that flashes, aiming at businesses who want to be seen in their community and support a growing source that quite frankly beats the local council's own site.

What I'd Tell You If You're Building Something Similar

  • Git makes a fine database for slow-changing public data. You get history, diffs and rollback for free, and dates from git log.
  • Don't write a file that hasn't changed. It keeps commits, builds and "last updated" dates honest.
  • Don't trust a scheduler that won't tell you it's late. A Raspberry Pi and a free Worker beat it.
  • Generate pages from data, but write the important ones by hand. The generated pages are useful long-tail; the hand-written intros are why people come back.
  • Test the structure, not just the code. Most of my bugs were a file in the wrong list.

If you live in or around Cheltenham, have a look at the site or sign up to The Cheltenham Week Ahead, a free Monday email. If you're building something like it for your own town, I'd love to hear about it - maybe a register/webring. I write about this and other side projects on my blog.

Top comments (0)