This is a submission for the Kaggle Benchmarking Challenge.
I work with AI video tools, and the part that keeps breaking is not the pretty single shot. It's continuity. You settle on a character, a location, a prop, and three steps later the model has quietly forgotten one of them. So instead of asking "which LLM writes the best story?", I asked something a film production actually cares about:
Can an LLM keep creative control of a film as it moves from idea to shot list, while a director keeps changing their mind?
What I Benchmarked
Each model gets a one-line premise with a few hard rules, then plays the whole crew in one conversation:
- Writer: a three-act treatment (exactly N characters, a limited number of locations, a list of banned words, one required story element)
- Casting director: one entry per character
- Production designer: sets, props, colour palette
- Director of photography: a shot list where every shot names its set, characters and props
Then the director gives three notes, one after another. None of them spell out their consequences:
- Era shift: "Move the whole story to 1890s London." Every phone, laptop and fluorescent light has to disappear, everywhere.
- Budget cut: "We lose the set X." Every scene there has to move, and the production should end up with one set fewer.
- Merge: "Characters A and B become one character, C." Neither old name may survive.
After each note the model returns the full cast, sets and props, but only the shots that change, the way a real change request works. Shots it doesn't mention carry over untouched. So the model has to work out for itself which shots a note affects: miss one, and the old prop or character name is still sitting in it.
There are 12 film ideas (a pharmacy noir, a lighthouse logbook from the future, a heist by retired museum guards, and so on), split into two Kaggle tasks of 6, Creative Control A and B. Each half's score is Kaggle's own leaderboard number; the overall score is their average.
How it's scored. Most of it is checked by plain code, not by another AI:
| Part | What the code checks |
|---|---|
| Constraints | Cast size, number of sets, banned words, the required element |
| Consistency | Every character, set and prop in the shot list actually exists in cast and design |
| Director's notes | Each note counts only if every consequence landed, and nothing crept back later |
| Story quality | The one subjective part: a judge model answers five yes/no questions about the treatment |
A note that is half-applied counts as not applied. A laptop in 1890s Vienna is not a 5% problem; it's a reshoot.
Two rules I held to throughout:
- Odd formatting is not a creative failure. Models wrap their JSON in comments, capitalise field names, put their reasoning before the answer, forget to escape a quotation mark. I read all of that leniently. Only missing or garbled content counts against a model.
- Each step must be answered within 16,000 tokens, about twice the longest answer any model gave. A crew member who never hands in the story outline has lost control too.
Models Tested
I picked a big and a small model from several labs, plus open models, to ask whether paying for the flagship buys you creative control.
| Model | Overall | A / B | Director's notes | Story quality |
|---|---|---|---|---|
| GPT-5.5 | 0.94 | 0.96 / 0.92 | 0.94 | 0.85 |
| Gemini 3.7 Flash | 0.90 | 0.89 / 0.90 | 0.85 | 0.73 |
| DeepSeek R1 | 0.88 | 0.87 / 0.90 | 0.83 | 0.70 |
| Gemma 4 31B | 0.84 | 0.85 / 0.82 | 0.71 | 0.65 |
| Claude Sonnet 5 | 0.80 | 0.91 / 0.69 | 0.75 | 0.82 |
| GPT-5.4 nano | 0.76 | 0.74 / 0.77 | 0.40 | 0.70 |
| Claude Haiku 4.5 | 0.69 | 0.64 / 0.73 | 0.50 | 0.62 |
| GLM-5 | 0.64 | 0.55 / 0.74 | 0.75 | 0.69 |
| Claude Opus 5 | half A only | 0.43 / – | 0.67 | 1.00 |
Director's notes and story quality are averaged over the ideas a model finished. Claude Sonnet 5 lost one idea, GLM-5 three, and Claude Opus 5 three of its six to the 16,000-token limit: they think at length before answering, and on the era shift (which rewrites every shot) thinking plus answer ran out of room. A full Opus run also costs about $19, more than Kaggle's $10 daily model budget allows for one run, so it only ran on half A.
The two halves agree closely for most models (within 0.03), which suggests the scores reflect the model rather than the luck of which ideas it got.
Two models could not be evaluated: Qwen 3 235B and GPT-OSS 120B. Over several days, Kaggle's shared service for them was overloaded on most requests, even with retries and backoff. I left them off the table rather than score them on server load.
Findings
1. Following notes separates models far more than writing does
Story quality lands between 0.62 and 0.73 for every model except the three flagships. Director's notes range from 0.40 to 0.94. If you only read the stories, most of these models look alike. Hand them a revision and they come apart.
2. Every model loses control in its own way
- GPT-5.4 nano forgets the world moved. In 7 of 12 films, modern objects survived "move it to the 1890s": a laptop, a computer and a phone in 1890s Vienna, GPS in 1890s Norway, a refrigerator in 1890s Ohio. In 6 films they were still there after the final note.
- Gemini 3.7 Flash was the most reliable model on the board. Its one repeated "failure" turned out to be my fault (see finding 3).
- Gemma 4 handles the era cleanly and the cast clumsily. It never left a modern object behind, but in 4 films the merge shrank the cast wrongly.
- Claude Haiku 4.5 updates the shots it touches and forgets the rest. After "merge these two characters", the old names were still in the shots it didn't rewrite (5 of 11 films). After the era shift, it renamed props for the 1890s in the props list but left the old names in untouched shots.
DeepSeek R1 was close behind Gemini, with only stray slips (a neon sign in 1890s London, a fluorescent light in 1890s Ohio).
GPT-5.5 knows when to redo everything. On the era shift it rewrote every shot, the safe move when the whole world changes. On the budget cut and the merge it touched only the shots that needed it, and got them right. That's the job of a script supervisor, and it's why it leads the board.
Does paying for the flagship help? It depends on the lab. GPT-5.5 against GPT-5.4 nano is the biggest jump on the board: director's notes 0.94 against 0.40. Claude Sonnet 5 against Claude Haiku 4.5 is a smaller one, 0.75 against 0.50, and Sonnet's real gain is the writing: story quality 0.82 against 0.62. Claude Opus 5 wrote the best stories of all (a perfect quality score on every film it finished) but often couldn't hand in its work within the length limit. And the cheap Gemini 3.7 Flash and open DeepSeek R1 beat both Claude flagships on following notes.
3. When my rules contradicted each other, models picked sides
I made a mistake in my own brief. The opening rules say "between 2 and N sets". The budget-cut note says "we lose one set". In a film with exactly two sets, those two instructions can't both be obeyed, and models split cleanly:
| Model | Kept the original rule (built a replacement set) | Followed the director (dropped to one set) | Avoided the conflict (had designed three sets) |
|---|---|---|---|
| Gemini 3.7 Flash | 7 | 3 | 2 |
| Gemma 4 31B | 6 | 5 | 1 |
| GLM-5 | 1 | 6 | 2 |
| Claude Haiku 4.5 | 0 | 9 | 2 |
| DeepSeek R1 | 0 | 6 | 6 |
| GPT-5.4 nano | 0 | 4 | 8 |
| GPT-5.5 | 1 | 4 | 7 |
| Claude Sonnet 5 | 2 | 1 | 8 |
Gemini answered "we lose the dining room" by building a chef's office, and "we lose the lantern room" with a storm porch: the original rule wins, the director's saving never happens. Haiku and DeepSeek always side with the latest note. The flagships mostly avoided the conflict by designing three sets in the first place.
My check only accepted "drop to one set", which unfairly penalised the models that kept the rule. I kept the official Kaggle scores as they are, but re-scored the same saved runs accepting either choice: Gemini 3.7 Flash moves from 0.90 to 0.93, Gemma 4 from 0.84 to 0.87, the others by 0.01 or less, and the ranking doesn't change.
4. Everyone writes the same weak spot
For every model except Claude Sonnet 5, the judge's most common complaint was the same: "Each character wants something different, and those wants create conflict" failed, in 6 to 9 of 12 stories. LLM short films tend to have one character with a goal and several who exist to react. That's worth knowing if you use these models for story development.
5. Cast members vanish from the screen
A frequent consistency error was a cast member who never appears in a single shot. The model cast them, described their look, and then forgot to film them.
6. A second task: does the model think like a cinematographer?
The second task asks for camera direction on eight dramatic moments, each three ways, in fresh chats:
- Instructed: "A woman realises everyone at the dinner party knows her secret. Use a dolly zoom." Does it know the technique?
- Intent: "... Make the audience feel sudden isolation in a crowded room." Can it turn a feeling into a camera choice?
- Implicit: just the moment. Does it think of using the camera at all?
Each moment also names its stock staging (the alarm-clock close-up for "she wakes up late", the glance over the shoulder for "he's being followed"), and the judge checks whether the plan leans on it.
| Model | Overall | Avoids the cliché: told → left alone | Technique described correctly |
|---|---|---|---|
| Claude Sonnet 5 | 0.96 | 0.88 → 0.88 | 0.88 |
| GPT-5.5 | 0.96 | 0.88 → 0.75 | 1.00 |
| Gemini 3.7 Flash | 0.96 | 0.88 → 0.75 | 1.00 |
| Gemma 4 31B | 0.95 | 1.00 → 0.88 | 0.71 |
| GPT-5.4 nano | 0.95 | 1.00 → 1.00 | 0.62 |
| GLM-5 | 0.95 | 0.88 → 0.75 | 0.88 |
| DeepSeek R1 | 0.92 | 1.00 → 0.75 | 0.83 |
| Claude Haiku 4.5 | 0.91 | 1.00 → 1.00 | 0.67 |
Camera vocabulary is a solved problem. Every model plans filmable, concrete, motivated shots, so the overall scores sit between 0.91 and 0.96. That's a ceiling, not a ranking. I ran this task twice on identical prompts, and most per-model columns moved by 0.1–0.3 between runs, so I won't rank models on it. Two things held up in both runs:
- Left alone, models fall back on the obvious. Across all models, the plan avoided the stock staging 93% of the time when told which technique to use, but only 75% (first run) and 82% (second run) when given just the moment.
- Knowing a technique's name isn't knowing how it works. Claude Haiku was the weakest here in both runs. It described a split diopter as keeping one actor sharp and the other soft; a split diopter does the opposite and holds both in focus at once. Asked for "one long unbroken Steadicam take", Haiku and DeepSeek R1 planned four separate shots with lens changes in between.
Caveats
- 12 film ideas is a small sample. One missed note moves a model's notes score by about 2%, so treat close scores as ties.
- The judge is Gemini 3.8 Flash. It may favour Gemini-family stories, so I'd read the quality column with care. The director's-notes and consistency columns are checked by code and aren't affected.
- Infrastructure noise. Kaggle's shared model service sometimes refuses requests under heavy load. The benchmark retries with backoff, but an idea that still fails scores 0.
My Benchmark
Creative Control: LLMs as a Film Crew on Kaggle
Kaggle's overall leaderboard averages all three tasks, so it places the models a little closer together than the film-pipeline tables above: the camera task is near its ceiling for everyone. The order is the same at the top: GPT-5.5 (0.94), Gemini 3.7 Flash (0.92), DeepSeek R1 (0.90).
The benchmark has three Kaggle tasks, each with its code, notebook and every model's saved conversations:
- creative-control-a and creative-control-b: the film pipeline above, 6 ideas each
- camera-storytelling: the camera task
You can read exactly how each model handled each note.
Top comments (0)