When I started building Clusterflick, I thought I was writing a scraper. Pull some listings, tidy them up, put them on my calendar. I'm a software engineer, and that's a software engineering job.
A couple of years and 450+ venues later, I've noticed that a lot of my time goes on things that aren't really software engineering at all. Some weeks I'm a sysadmin, some weeks I'm reading up on data licensing, and some weeks I'm trying to work out how 70mm and IMAX relate to each other.
In a company, most of these would be separate people with handoffs between them. On a side project the handoffs disappear, and so does anyone telling you that you've spent three weeks wearing one hat and ignored all the others. So here are the hats, roughly in the order the data flows through them, from the machines at one end to the people at the other.
Before Any Code: Researcher and Film Nerd
The first job happens before anything gets scraped: working out what's actually out there to scrape. London has the big chains and the obvious independents, but it also has film clubs in pub back rooms, pop-up outdoor screenings, festivals that run for a week a year, and churches that screen silent films with a live organ accompaniment 🎬
Finding these is ongoing research. New venues open, film clubs start up, festivals announce their programmes, and none of them send me an email about it.
Close behind is a hat I didn't expect to need: film knowledge. I know what I like to watch, but listing screenings properly means understanding what each one actually is. Film formats were a real learning curve: 70mm is a film source format, IMAX is a presentation format (and a brand), and 70mm IMAX is its own thing again (technically another source format), with only a handful of screens that can show it. Venues don't always say which one they mean, so working out the relationship between them came before I could model any of it.
👆 As Nolan Intended
Accessibility is harder, and I'm still not sure I've got it right. Subtitled, subtitled for the hard of hearing, captioned, open captions and closed captions overlap in ways that aren't obvious until you try to represent them as data. I'd be lying if I said the output models them perfectly.
Getting the Data: Scraper, Prompt Tuner, QA
Scraping specialist
This is the hat I started with, and it's grown the most. Three venues with a simple fetch-and-parse became half a dozen ticketing platforms, venue-direct sites, and a growing collection of sites that make this difficult. Bot detection is its own discipline: stealth plugins for Playwright, a Camoufox pilot for the source that blocks everything else, and a lot of patience.
The BFI is the clearest example. Scraping Southbank, IMAX and the festivals takes about two hours, run sequentially because of rate limiting, and the bulk calendar can't be trusted to include everything, so it has to go page by page.
👆 Runner times for BFI; red is the Pi 4, blue is the MacBook Pro
Prompt tuner
Matching a listing like "Late Nights: The Thing (1982) on 35mm + Q&A" to the right film is where LLMs come in, and tuning them has become its own job. Prompts get tested against known data, AI model providers get compared, and sometimes the right answer is to stop using an LLM. Listing categorisation moved to Jev in September, with the LLM kept as a fallback.
The lesson that cost me the most was about providers. When Gemini started throwing persistent 503 capacity errors, the tempting fix was a live fallback to another provider. But then the same input could go through different models depending on the time of day, and I'd lose confidence in the output because I couldn't tell which model had produced it. Now if I change provider, I switch over completely and retune the prompts for the new model, rather than having several models in play at once.
QA
Title normalisation has a test suite of over 25,000 labelled titles, and a daily CI job opens pull requests for new titles that need reviewing. Reviewing those PRs is manual QA in everything but name, and it's the hat I wear most often without thinking of it as one. I did try automating it, and it wasn't worth it.
The same goes for the listings themselves. Nobody emails me when a screening is wrong, so the only way I find out is by spot-checking the site against the venues.
Keeping It Running: Sysadmin, Hardware Tinkerer, Security
All of this runs on GitHub Actions, partly on a fleet of Raspberry Pis and a headless MacBook Pro acting as self-hosted runners. Someone has to keep them registered, updated and online, and diagnose why one has dropped off the network at 6am.
That's the sysadmin hat, and it comes with a more physical cousin: the hardware tinkerer. The Pis originally ran from SD cards, and the CI workload (npm installs, git checkouts, Playwright downloads) writes something like 25–35 GiB a week. SD cards don't enjoy that, and they noticeably degraded 😅 Moving everything to USB-attached SSDs fixed the wear problem and introduced a new one, involving TRIM, a particular USB-to-SATA adapter, and a kernel that occasionally decides to switch TRIM off again.
There's a security hat too, though it's lighter than you might think. The Cloudflare Workers in front of TMDB exist to keep an API key out of the browser, and Content Security Policy on a static Next.js export turned out to be more awkward than expected. For a long time the threat model was simple (no logins, no sessions, scraped text rendered through React), which helped me decide how much effort that deserved. Adding accounts through Firebase changed that, since Firebase handles the auth but there's now something worth protecting in the browser, so this hat is due another outing.
The Hat Under All the Hats: Software Engineer
General software engineering doesn't sit at one point in the pipeline, because it's underneath all of it. Most of my engineering effort goes into making the rest of the hats easier to wear: adding a new venue should be configuration rather than code, a broken scraper should fail loudly but have a contained blast radius, and the retrieve/transform split means I can re-process data without hitting a venue's website again. Getting the data model right early on, and leaning on GitHub as infrastructure, are what let one person keep it all going.
Shaping and Sharing: Data Steward, Licensing, Analyst
At some point other people started using the data. filmhose.uk, culturecity.london and everycinema.london all build on Clusterflick's releases, and that turned me into a data steward. Schema stability became a personal rule, because a change that's harmless for my site could break someone else's.
With other people using the data came a hat I never expected to wear: licensing. A third-party request to use the data made me work through what I could actually license. The published data is factual scheduling information, I have no agreements with venues, and the result was CC BY 4.0 on the public datasets and a data licence page explaining the position.
Then there's the analyst. The manage interface tracks stats over time, and it's where most of my performance work starts. The site build used to take over 30 minutes and now takes around 8, because a lot of data is pre-processed up front. The transform step dropped from around 25 minutes to 14 after I changed how source matching works for the 20+ sources spread across 450+ venues.
The number I watch most is the film match rate, which usually sits around 92–94%. More matches is better, but finding out why something didn't match is hard work. It could be a normaliser issue, a film missing from TMDB, or bad data from the venue, and each one needs a different fix.
Some of the analyst work is checking that things I built actually do what I think. The LLM calls are cached daily, so re-running transform should be nearly free, except it wasn't. Changes on the TMDB side, like a film's popularity score shifting, were busting the cache for some of the most expensive calls, and I only found that out by looking at the hit rate.
👆 The second run of the day is significantly cheaper, with a 90%+ cache hit rate
In Front of People: Front End and Product
The front-end developer hat is the most familiar one from my day job. The site is a Next.js static export, with filters, a calendar view, a planner view, and now accounts, and the main challenge has been loading 30,000 showings without the browser falling over.
The product owner hat is harder. Deciding what to build next means weighing a user profile page against Letterboxd sync against new movie alerts, all competing for the same evenings. The best tool I have is dogfooding: I use Clusterflick to decide what to see, and the features that annoy me are usually the ones that annoy everyone else. That's where filters in the URL came from, so I can link to a specific view and the back button takes me to the one before. It's also why there's now a planner page to see a day's showtimes at a glance, and why the festival pages now have a timeline showing how the festivals overlap.
Alongside it is a small designer hat (a brand palette of neon pink, electric blue and buttery yellow, and a lot of fiddling with layout), and a finance hat that turns up wherever there's a bill: paying for the Gemini tier, comparing providers partly on cost, buying SSDs.
Telling People: Marketer, Writer, Speaker
These are the hats I'm newest to.
Marketing and SEO are where I'm least comfortable. I've been learning Bing's webmaster tools, getting pages indexed, building a few backlinks, and planning some curated reports. Most of it is trial and error, and I suspect a lot of what I'm doing is obvious to anyone who's done it professionally.
👆 I did buy stickers though, and that's half of marketing, right? 🎟️
Writing has been easier, and this series is the result. Writing up a decision forces me to understand why I made it, and more than once a post has turned up something I wanted to change.
The hat I'm trying on next is speaker. I submitted a talk, "Cutting Room Floor: Wrangling London's Cinema Data", to NIDC this year. It wasn't accepted, but the feedback was useful, especially the suggestion to talk properly about the ethics of scraping public listings. I'm reworking it for a local meetup next.
Picking a Hat
Counting them up, that's well over a dozen roles for a single cinema listings site, and I don't think I have a favourite. The fun is that I can usually pick a hat to suit what I feel like doing:
- If I want more data in the system, I put on the researcher hat and go looking for new venues.
- If something about the site bugs me, I put on the front-end and product hats and make it better.
- If something's frustratingly slow, I put on the analyst hat and find out why.
The hats I like least are probably sysadmin and scraper, and only because I usually put them on when something's gone wrong: a runner misbehaving, a workflow failing, a venue that's changed its site overnight.
I'm proud of this project, and I get a lot of satisfaction from working on it. But the best hat I get to wear is the one at the very end of the pipeline: end user, planning my next trip to the cinema 🍿
Top comments (0)