A listing arrives from the Cinema Museum in Kennington. It's the quarterly meeting of the UK Buster Keaton Society, who are also known as The Blinking Buzzards, it runs from 4pm to 7pm, and members will be requesting beloved gems to watch together.
So what is it? It's a club meeting, so is it an event? It's an appreciation society discussing a comedian, so is it a talk? It's a programme of silent shorts, so is it a shorts programme? When I reviewed those listings myself, I marked one as multiple-movies and gave up on the next, recording it as having no single right answer. Even for a human this is a hard problem to solve.
Clusterflick pulls listings from 400+ venues across London, and most of them match a film in The Movie DB so there's nothing to decide; they have to be movies because they matched a movie. The leftovers need sorting into one of ten categories (movie, tv, quiz, comedy, music, talk, workshop, shorts, multiple-movies, event), which is the enum I settled on a while back when it became obvious that modern independent cinemas don't just show movies. Somewhere between 450 and 600 listings go through that decision every day.
For the last year the job has belonged to a language model: title, runtime and description go in wrapped in a prompt, and JSON comes back with a category, a one-line reason and a confidence score.
{
"category": "movie",
"confidence": 9,
"reason": "This is a single feature film screening.",
"title": "Star Wars: Episode VI Return of the Jedi (1983)"
}
💡 The title on TMDB is "Return of the Jedi", which is why this longer title didn't match and we needed categorisation. Just another fun task for the normaliser to deal with.
Then I came across Jev, which isn't a language model. You hand it structured state and a set of typed questions, and it returns a probability distribution over the possible answers, with no text generation anywhere in the loop. Given that my problem is picking one of ten options and I throw the prose away unread, it seemed worth an evening to find out whether it could do better than a prompt I'd been hardening for a year.
What changed when I moved to Jev
The reason field is no longer needed. I've written before about asking the model to explain itself, which made it noticeably more cautious: the difference between blurting out an answer and having to show your working. That improvement was real and it still holds for the LLM path. It has nothing to attach to here though, because Jev doesn't produce prose in the first place, so there's no explanation to demand and no fluent wrong answer for it to talk itself out of.
The things it used to reason out loud about can be asked directly. On its way to a category, the old prompt is really working out a handful of underlying facts: is there a feature-length film here, is there more than one, is this television, is a film actually screened rather than just discussed. Jev lets me ask those as four separate yes/no questions alongside the category question, and they come back in the same round trip at no extra cost, because it evaluates every question in a request in parallel. They also land on exactly what it's documented as weakest at, which is counting and runtime arithmetic, so asking each one narrowly is the recommended way to handle those.
Category descriptions became structured objects. Several of my categories are defined mostly by what they exclude, since a concert film is a movie rather than music, a screening of a comedy is a movie rather than comedy, and a film with a Q&A afterwards is still a movie rather than a talk. The docs are blunt about why that matters, because Jev "answers the question you wrote, not the one you meant", with scoping words and negations read at face value. A language model will usually infer what I was driving at from a loosely worded description, and Jev won't, so a "not a comedy film screening" clause buried in a sentence does less work than the same exclusion given a field of its own. Each option now carries an explicit excludes list, and the advice is to reach for that only once two options are similar enough that the model keeps confusing them.
The arithmetic moved into my code. Jev is documented as not being a calculator, and it does better on semantic language than on numeric formats. So instead of handing over Duration: 73 minutes and hoping, I do the comparison myself and pass the result as a fact. It's phrased deliberately without naming a category, so it states the relationship and leaves the conclusion alone.
The whole programme runs 73 minutes, shorter than the 80
minutes a single feature film usually runs.
Measuring it properly
I hand-labelled listings until I had 142 of them, including two held-out sets that hadn't been used to tune anything, then ran both categorisers across a full release and adjudicated by hand every case where they differed.
Where they disagreed, Jev won comfortably. Out of 434 listings across 111 venues, the two gave different answers 102 times. Jev was right on 73 of those, the old path on 18, and 11 were genuinely ambiguous.
On the hand-labelled set, Jev got 128 of 142 right, or 90%. Adjusting for how often each confidence level actually turns up across a release, that works out at roughly 93% for Jev against 82% for the LLM path. It also runs at a 277ms median and costs about 2.4x less.
I kept the hand-labelled listings. Every row carries its own copy of the title, runtime and description, and records the answer I judged correct rather than what either model said. When this is iterated on in the future, it can be used to score whatever I try next.
The threshold I didn't carry over
The old categoriser gated on self-reported confidence: the model returned a 0-9 score and anything at 7 or below got thrown away and filed as event.
That wasn't arbitrary. It came out of a real problem I'd hit and written up in January, which was too many confident wrong answers, where the model picks something and reports high confidence even when it's clearly a stretch. If the score is unreliable at the top, leaning on it at the bottom is a reasonable thing to try.
So the obvious move was to do the same with Jev, whose confidence is a calibrated probability rather than a number the model picks for itself. Measuring it first is the only reason I didn't:
| gate | correct |
|---|---|
| no gate | 128 / 142 |
| 0.60 | 117 / 142 |
| 0.85 | 85 / 142 |
The flaw is in what happens to a discarded answer. Throwing one away doesn't leave the listing unclassified, because it still has to go on the website under something, and that something was event. So a low-confidence guess of shorts gets replaced by a confident published claim that this is not a film at all, and that claim is wrong more often than the guess it replaced.
That measures gating Jev, not the old LLM threshold, which I never measured and have now retired without ever finding out what it cost. There is a hint in the prompt though, which had quietly grown the lines "Do not let combined runtimes reduce your confidence" and "score it 8 or 9 so the choice isn't discarded as low-confidence". I'd been writing instructions to get answers past my own filter. 😅
Most of the win is refusals it doesn't make
Go back to those 102 disagreements, because one number in them explains the whole result. Of the listings where the two differed, the old path had published event on 51 of them, and Jev never once moved a listing into event that the LLM path had managed to categorise.
So the improvement isn't mostly better judgement about hard listings, it's that fewer listings get abandoned. Between the threshold discarding answers and the model itself reaching for event when a listing looked awkward, roughly half the disagreements were cases where the old path had declined to commit and Jev simply answered.
What happened in production
It went live yesterday. Comparing the first Jev release against the previous one, joined on showing ID across the 6,472 listings that appear in both:
-
eventfell from 128 to 29, and the 29 left are things like Games Night, a cheese and port afternoon, and an awards ceremony, which are genuinely uncategorisable -
multiple-moviesrose from 59 to 90, mostly double bills the old path had given up on - only 2.41% of listings changed category at all, so nothing lurched
- categorisation cost dropped 55%, from $0.0767 to $0.0342 per run
- 535 calls went out and not one fell back to the LLM
The whole thing sits behind a CATEGORISER environment variable so I can put it back in seconds, and the language model is still wired up underneath as a fallback for timeouts and rate limits. That fallback logs loudly when it fires, because a silent one would look exactly like a working Jev run.
What it still gets wrong
The misses I find interesting aren't really about model quality.
The Nickel runs mystery screenings, where the film isn't named until you're in the room. The listing for one of them opens "Mystery Erotic Thriller presents undisclosed deep cuts from the genre's overheated history", and the description goes on in the plural about neo-noir, studio sleaze and direct-to-video delirium. That's a description of the strand, not of the single film that's on tonight. Jev reads the plural and says multiple-movies. I already have a guidance line warning about exactly this trap, telling it to count only the films a listing actually names, and it still loses to a description written entirely in the plural.
"450 XL: A silent movie" has the words "a silent movie" in the title and is not one: it's a live performance. Jev said movie at 0.24 confidence, which is the system working exactly as intended and still being wrong. It knew it didn't know.
"Supernatural Mini Marathon" came back as multiple-movies when the right answer is tv. The description the venue supplies reads exactly like a film plot and contains none of the signals I listed under tv: no broadcaster, no "episode", no "series". What the listing does have is "Marathon" in the title, and "Marathons" is one of the examples I wrote under multiple-movies. Answering correctly means already knowing that Supernatural is a television programme, which is knowledge about the world rather than anything the venue wrote down.
The one genuine gap is mine: three single films of 60 to 73 minutes landed in shorts. My criteria cover "a single film under 40 minutes" and "several films, none reaching 80", so a lone 65-minute documentary fits neither while I'm actively telling Jev it runs "shorter than the 80 minutes a single feature usually runs". I never defined that case, so it picked the nearest thing that fit.
Where this kind of thing fits
Jev is new enough that the useful question isn't whether it beat my prompt, it's what shape of problem suits it. After one evening with a real one, my rough test is three things: the output is one of a fixed set of options rather than prose, the decision is small but happens constantly, and you could sit down and hand-label a few hundred real examples without losing the will to live. Categorising cinema listings hits all three. Summarising them wouldn't, and neither would anything where I actually wanted the sentence it wrote.
Top comments (0)