DEV Community

Cover image for Outgrowing the Raspberry Pi Cluster
Alistair
Alistair

Posted on

Outgrowing the Raspberry Pi Cluster

There used to be six Raspberry Pi 4s on a shelf in my living room. I wrote about them earlier this year: cheap, low-power self-hosted GitHub Actions runners that give Clusterflick a residential IP address for the cinema websites that block GitHub's own runners.

They've now been replaced by two Dell OptiPlex Micros on top of my wardrobe, running 16 runners between them on a 5G connection of their own. Here's why the Pis stopped being enough, and what changed along the way.

The power supply matters more than you'd think (again)

In the last post I said the power supply matters more than you'd think, and it bit me again. One of the Pis started dipping below the voltage it needed at nearly every job start, and fell out of the pool for around five hours at one point. Another developed problems with its SSD, so with two of them out, six runners were suddenly four.

Ten runners wasn't enough

Six runners had never quite been enough, which is how my old MacBook Pro ended up running four more. That took the total to ten, but it also meant my laptop was a permanent part of the infrastructure.

The retrieve workflow has 15 jobs that need a self-hosted runner, and they all start at the same time. With ten runners, five of them queued on every run, and GitHub Actions has no way to say which jobs should go first.

That matters because the jobs are very different lengths. Close-Up Film Centre takes under five minutes, while the BFI takes over an hour because it has to go slowly to stay within rate limits. If BFI ended up at the back of the queue, it waited for five other jobs to finish before it could start, which was around ten minutes in the worst case and longer on the Pis. BFI is the longest job in the run, so every minute it spent waiting was added to the whole pipeline.

So the goal was enough runners that nothing queues. Getting the MacBook back, retiring the Pis and having a spare runner for the hourly health checks were all nice to have on top of that.

How the jobs are split

Those 15 jobs cover a lot of venues, and how they're grouped is deliberate:

  • Each chain runs in its own jobs, never mixed with other venues. Cineworld, Curzon and ODEON get one job each, and Vue and Picturehouses are split in two because each was slow enough to hold up the run on its own. If a chain's site changes or starts blocking requests, only that chain's job fails.
  • The sources that fail most often are kept apart from the steady ones, so re-running a failure doesn't repeat everything else with it.
  • Sites with aggressive bot detection are never hit from two places at once. BFI runs in a single job, one request after another.
  • Where a venue runs depends on the venue. Some block GitHub’s IP ranges and some need Camoufox, so they run self-hosted; everything else runs on GitHub’s runners, which keeps the self-hosted count at 15.

This is part of keeping Clusterflick's footprint small, which I wrote about in my post on the ethics of scraping. Grouping by site means I always know where the requests for a given site are coming from, and that they're never doubled up.

Why not more Pis?

In the last post I said six Pis at £50 each is £300, which got me six runners. The two OptiPlexes were bought used for a little over £200 each, and together they run 16.

Space was the other problem. The Pi cluster sits in an IKEA SMARRA box, which is 30×30×23cm. That's a nice contained size, but scaling it up 3x would be unwieldy. The two OptiPlex Micros take up about the same space.

I also previously said that each Pi being independent meant one playing up only took one runner offline. And that's why I went for two OptiPlex machines rather than one bigger one: if either goes offline, the other can still process every job, just more slowly.

The two machines are an OptiPlex 7080 Micro and an OptiPlex 5090 Micro, both with 16GB of RAM. Because they were bought used, I tested each one before trusting it, with SMART checks on the drive, stress-ng for CPU and temperatures, and memtester for the RAM. They run Ubuntu Server 24.04, with the runners set up much like the Pis were: each in its own directory and service, sharing one cache. The setup is documented in the self-hosted-workflows repo.

I'd planned on five runners per machine, based on how much memory I expected the browser-based jobs to need. Real runs used much less than that, so each box now runs eight with room for more if I need them.

On speed, they're faster than the Pis and slower than the MacBook. BFI now takes about 1h 15m, and a full retrieve run takes about 1h 20m.

Subset of retrieve job durations by runner 👆 Job durations on the Pis (orange), the MacBook (blue) and the OptiPlexes (pink and purple)

A network of their own

The runners are now on their own 5G connection, a ZTE MC889 outdoor unit on a Smarty SIM, separate from my home network. Clusterflick's repos are public, and GitHub advises care with self-hosted runners on public repositories, so keeping them on their own network is a sensible precaution.

A mobile connection means a shared IP address behind carrier-grade NAT (no longer residential, but still not a datacentre range), which cuts both ways. Its reputation is worse, which caused issues with a few venues' firewalls, so those moved to GitHub's runners. On the other hand, my requests come from the same address as lots of other people's, and getting a new address is as simple as restarting the router, which I can do remotely from a workflow.

It's also less stable than a fixed line. Evening congestion has been enough to fail the hourly health checks, and a retrieve could fail the same way. The retrieves run at 4am and 4pm, which I picked to stay away from the times the cinema sites are likely to be busiest, such as evenings and late mornings. That just happens to avoid the worst of the network congestion too.

Daily bandwidth usage 👆 Bandwidth since the connection went in on 19 October

An advantage of having a dedicated connection is that I can track the bandwidth usage. Each retrieve uses ~5GB, so two a day comes to around 300GB a month. Each set of health checks, run hourly, is about 300MB which adds roughly another 200GB. So the connection needs a little over 500GB a month in total.

Hourly bandwidth usage 👆 Bandwidth for the last 48 hours

What happens to the Pis

One will become a Pi-hole for my home network and another a Pi-hole for the Clusterflick network. On the runner side, that shows me which requests the scraped pages make, and lets me block the ones that aren't needed to speed things up. I'd rather do that than filter requests inside the Playwright setup, because a Pi-hole can be tuned or turned off without touching the runners. A third might run a dashboard, and I'll probably sell the other three to recover some of the cost.

Six Pis in a box got Clusterflick a long way. Two small PCs on a wardrobe cost a bit more, but it's a project I enjoy, and I'm happy to keep investing in it.

Top comments (0)