In October 2024, Simon Willison gave sixteen LLMs the same eight words:
Generate an SVG of a pelican riding a bicycle.
He has been running it on every new model since. Twenty-three months later there are 103 of these drawings scattered across dozens of his posts, and he has written a section explaining why you should not use his benchmark to compare models.
We pulled all of them into one page — pelicanzoo.ai — because reading them in order turns out to be a surprisingly good history of the last two years. This post is what we learned putting it together, and how the thing is built.
Everything here is Simon's work. The images, the commentary and the judgement calls are his; we mirrored and organised them, and every specimen links back to the post it came from.
The prompt is deliberately unfair
His stated reasons are mundane: he likes pelicans (he lives in Half Moon Bay, they're on the beach), and he was confident no SVG of a pelican on a bicycle existed in any training set.
The reasons it works are better. LLMs are text models that can't see, but they can write code, and SVG is code. So the task is to assemble a picture out of coordinates and radii with no feedback loop — the model never sees what it drew. Bicycles are hard for humans to draw from memory (try it: you'll get the frame wrong). And, in his words:
Pelicans can't ride bicycles. They don't have the right shape.
There's a bonus that matters more than it sounds: models leave comments in the SVG. <!-- wheel -->, <!-- beak -->. You can read what it meant to draw and compare it against what came out. The gap is usually the joke.
2024: it's mostly shapes
The early attempts are bad in specific, funny ways. Claude 3.5 Sonnet produced two circles, a triangle, and a beak Simon described as a yellow banana smile. Llama 3.3 70B drew a small circle, a vertical line, and something shaped like a sink. Qwen2.5-Coder's bicycle is a pile of brown shapes that reads as a tractor. Mistral Small 3 delivered a squat white duck squatting on a barbell.
Then DeepSeek R1 landed in January 2025, and shortly afterwards Nvidia lost about $600bn in market cap. Simon later singled its pelican out as "the pelican that crashed the stock market" — the best one so far: a recognisable bicycle, a bird next to it that's arguably a pelican, not actually riding it.
For that first year the drawings tracked real capability fairly well. Models that drew better pelicans were better at writing code. A joke about how silly model comparisons are started behaving like a real benchmark.
The 560-match tournament that cost 18 cents
By June 2025 he had 34 pelicans and wanted a ranking without looking at 34 images himself.
So he had Claude write a tool that renders any two pelicans side by side and screenshots the pair. 34 images, every pairing, 560 matches. Each match went to GPT-4.1 mini with instructions to pick the better "pelican riding a bicycle" illustration and explain why. Elo scores out of the results.
Total judging cost: about 18 cents.
Winner was a Gemini 2.5 Pro preview; last place, Llama 3.3 70B. The judge's reasoning for that matchup: the left image clearly depicts a pelican riding a bicycle, the right is a few minimal shapes that don't read as anything.
The same talk includes the cost comparison that stuck with me. o1-pro: 88.755 cents for one pelican. Gemini 2.5 Pro: 4.77 cents. Gemini's was better.
The models start editorialising
August 2025, Simon runs Qwen3-4B-Thinking locally. It declines to draw a bicycle. It returns a blue circle with red text across it:
This is art - pelicans don't ride bikes!
He made that the headline of the post.
In November, Gemini 3 at low reasoning effort gave its pelican a little hat, with an SVG comment reading hat (optional fun detail). At high effort the frame geometry was finally correct.
He decided the task had gotten too easy and escalated: a California brown pelican, bicycle with spokes and a correct frame, large gular pouch, visible feather texture, unambiguously pedalling, in breeding plumage. He attached a photo he took himself.
Most models missed the detail that makes it a trick question — the California brown pelican isn't brown.
The laptop model that beat Opus
April 16, 2026: Qwen3.6-35B-A3B and Claude Opus 4.7 shipped the same day. Simon ran Qwen as a 20.9GB quantised model on his MacBook and Opus through the API.
He gave it to Qwen. Opus got the frame geometry wrong, and got it wrong again at maximum reasoning effort.
This result made him suspicious of his own benchmark. There's a persistent theory that labs train against his "stupid test", and he'd been keeping a spare prompt in reserve for exactly this moment. He burned it: a flamingo riding a unicycle.
Qwen's flamingo wears sunglasses and a bow tie, appears to be smoking, and is flanked by heart emoji and the caption "Flamingo on a Unicycle". The source comment says Give the flamingo sunglasses!. Opus drew a competent, entirely unremarkable flamingo. That one went to Qwen as well.
His own framing was careful: he has enormous respect for Qwen, and he does not believe a 21GB quantised model is more useful than Anthropic's latest flagship. But if the thing you want is an SVG of a pelican on a bicycle, the laptop won that day.
That's roughly where pelican quality and model capability stop tracking each other.
The inventor says don't use it
Kimi K3, July 2026. Its pelican burned 13,241 reasoning tokens and cost 25 cents. Simon used the post to write a section on what the benchmark is still good for.
His conclusion after 21 months: it was never a good benchmark. The correlation in year one was a surprise and it's basically gone now — GLM-5.2 draws a better pelican than GPT-5.6 or Claude Fable 5, and he doesn't think GLM is in that class. More importantly it tests nothing about what actually matters today, namely whether a model can hold a long conversation and call tools reliably. His words: don't use pelicans to compare models.
So why keep going? Two reasons, both practical:
- It forces him to actually run each new model. Posting a pelican is proof he got the thing working, rather than just reading the launch post.
- Even one prompt leaks information. On the Kimi K3 run he noticed his eight-word prompt was billed as 95 input tokens, and sending just "hi" cost 86 — implying about 85 tokens of hidden system prompt.
September 1, 2026: Claude Fable 5.1, across five reasoning levels. Low and medium barely think and finish in about twenty seconds. Maximum ran 13 minutes 54 seconds, emitted 65,927 tokens, cost $3.30, and produced a pelican in a blue cap with a fish in the basket. Best pelican he's seen from Anthropic.
The benchmark has become a ritual rather than a measurement. It won't tell you which model is strongest. One pelican's invoice and reasoning trace will still tell you a lot about how a model works, and whether it'll decide on its own to put a hat on your bird.
Building the zoo
The drawings live across two years of posts, which makes reading them in sequence tedious. So: one page, 103 specimens, 66 models, 12 vendors, October 2024 to now. Each one is tagged with model and date and links back to Simon's original post.
A few things that turned out to be more interesting than expected:
24 of them are still vector. Where the original SVG survives, the page serves the SVG — zoom in as far as you like, and four of them are animated (wheels turning, pelican bobbing). The rest exist only as screenshots from his conference slides, so they're labelled print rather than live. Making that distinction visible mattered more than we expected; a screenshot of an SVG and the SVG are not the same artifact, and the ones that are still code are the ones you can actually inspect.
The critic reads source, not pixels. There's a button under each specimen that asks a model to review the drawing — and it's given the SVG source, not an image. Half of that is cost. The other half is that it's funnier and more specific: it can complain that the body is an ellipse of radius 104, or that the rear wheel is out of proportion with the frame. Using models to review models closes a loop that probably shouldn't be closed.
Submissions have no backend. You can paste in an SVG a model drew for you. The page sanitises it in the browser, then generates a pre-filled GitHub pull request. No server, no database — one pelican is one file. It's a static site on Cloudflare.
The sanitiser is the part I'd actually defend in review. Model-generated SVG pasted in by strangers is untrusted markup, so it strips <script>, <foreignObject>, iframe/object/embed, inline on* handlers, javascript: URLs, @import, and any remote href/src/url() reference. Data URIs stay — they're self-contained. Remote references go because they leak the visitor's IP to whoever hosts them and let someone swap the picture out later.
Two design decisions in there I'd repeat:
// It reports what it removed instead of only returning a cleaned string.
// @returns {{ ok, svg, removed: string[], error }}
The same function runs in three places — the static build, the browser preview, and CI on incoming pull requests. Because the site strips before it builds the PR, a submission that still arrives dirty was hand-crafted, so CI stops it for a human to read rather than silently cleaning it. A sanitiser that returns a clean string and nothing else throws away the signal that someone was trying.
The other one is dull and bit us first:
// Percentages need something to be a percentage of; without a viewBox an SVG
// whose width/height we strip will stretch to fill whatever box it lands in.
We strip fixed width/height so specimens scale in a grid. Do that to an SVG with no viewBox and it stretches to fill its container. So the sanitiser synthesises one from the original dimensions when it's missing.
There's also a guessing game — you get a drawing and four model names, 73 rounds over 51 models. Distractors are picked from different vendors on purpose. Three Gemini variants side by side isn't difficult, it's just unfair. The 2024 ones are easy. The 2026 ones are basically a coin flip, which is its own kind of result.
Credit
The images and the original keeper's notes are Simon Willison's, collected from his posts, and the footer links back to his pelican-riding-a-bicycle tag. We built the cage, not the birds.
If you've got one that's especially good, or especially wrong, the submission box takes it.
Sources
- Pelicans on a bicycle (2024-10-25)
- The last six months in LLMs, illustrated by pelicans on bicycles (2025-06-06)
- Trying out Gemini 3 Pro with audio transcription and a new pelican benchmark (2025-11-18)
- Qwen3.6-35B-A3B on my laptop drew me a better pelican than Claude Opus 4.7 (2026-04-16)
- Kimi K3, and what we can still learn from the pelican benchmark (2026-07-16)
- Claude Fable 5.1 made me a really nice animated pelican (2026-09-01)
Top comments (0)