Picture three SVG pelicans on bicycles. The first is beautiful (layered feathers, shiny wheels, a sunset behind it), but both feet hang above the pedals. The second is a rough sketch, but the frame connects, the body sits on the saddle, and the feet are actually on the pedals. The third looks fine as a still image. Press play, though, and one foot drifts off on its own path.
Which one should score highest? If you pick the prettiest, you're scoring taste. If you pick the reliable one, you're scoring whether the task got done. And if you only look at screenshots, you may never notice what's wrong with the third.
Here are the main ideas from our longer write-up on turning the "pelican on a bicycle" test into something you can re-check.
1. Write the task down before you argue about scoring
"Draw a pelican riding a bicycle in SVG" tests whether a model can turn a written request into code that renders the right picture. It doesn't prove anything about the model's internal spatial understanding. Don't add requirements at scoring time that weren't in the prompt. Split it into two kinds of test:
- Open task: keep it loose, and only penalize requirements that are clear and necessary.
- Constrained task: state the viewpoint, the visible relations (each foot touching its pedal), and the output format (one self-contained SVG, no external images or scripts) before anyone runs it.
Also, render-look-revise loops measure a workflow, not one-shot generation. Keep those results on separate leaderboards.
2. Don't rush to a total score; break "riding" apart
"Looks good, 8/10" is the easiest score to give and the hardest to re-check. Score the subject, the bicycle structure, the riding relation, and the presentation separately, and decide in advance what each one does not include. For example, feather detail shouldn't count toward "is it a pelican."
3. Rules, geometry, and visual judges each have a job
- The SVG DOM helps, but it's not a structural diagram that comes with the answer. Static rules work best for constraints that can actually be settled (self-contained file, no scripts).
- A geometric check has to define what it measures before it measures anything.
- Before a visual LLM judge takes the job, give the judge an exam. Ask it concrete questions ("is the left foot touching the left pedal?") rather than for a verdict on taste, and don't calibrate it on a single overall-agreement number.
4. Animation checks whether relations hold over time
It's not "a few extra screenshots." The question is whether the foot stays on the pedal while the pedal turns.
5. The scorer can be wrong too
A changed judge or rubric can make it look as if a model suddenly got worse. Items do go stale, but rewriting the test every day isn't more scientific. Change the constraint, not just the animal's name, and don't present a change in item difficulty as model progress.
What's actually worth automating is a procedure that can be audited later, and a leaderboard worth reading is one you can keep asking questions of.
The full article includes a sample constrained prompt, an acceptance table, and an example judge output record:
https://folkbench.com/en/blog/how-to-score-a-cycling-pelican?utm_source=luntan&utm_campaign=dev
Folkbench can be browsed without registering. Rankings and blog posts are open to everyone:
https://folkbench.com/?utm_source=luntan&utm_campaign=dev
How do you score open-ended generation tasks like this? I'd like to hear in the comments.
Top comments (0)