This is a submission for the Kaggle Benchmarking Challenge
What I Benchmarked
I've been writing design docs and decks at work, and I use LLMs to draw the architecture diagrams for them. The diagrams keep failing in one of two ways:
- they leave out an interaction that matters, or
- they include so much fine detail that someone new to the system can't see the design.
So I built archdiagram-bench, a benchmark for one question: given a real code repository, can a model draw a high-level architecture diagram that a newcomer could actually use?
The task
The model gets a real repository (BillTalk, a voice-operated billing app of about 80 files that I wrote), tools to list, search and read its files, and this request:
Draw an architecture diagram that explains the high-level design of this code base to a developer who is new to it.
- Show the end-to-end user flow from its starting point, for example an HTTP call or an interaction in an app.
- If a database or other data store is involved, show the read and write operations.
- Show every major part of the system and the interactions among them.
- Group the parts that run or are deployed together, for example agents or services on the same host, or everything that runs on the user's device.
It answers with draw.io XML, the format I'd actually paste into a design doc. I deliberately gave no box limit and no definition of "high-level". Judging the right level is the skill being measured, the same way I'd expect it from a colleague.
How a diagram is scored
- Real rendering. The XML is drawn by draw.io's own viewer in a headless browser, so the judge sees exactly what a person would see.
- A judge that knows the code. A judge model (Gemini 3.1 Pro) studies the repository once with the same file tools and keeps notes it reuses for every diagram.
-
Comparison with a human reference. The judge gets two images: the model's diagram and a reference I drew myself, about ten boxes. It marks:
- Closeness to the reference (40%)
- A checklist (30%): entry point, data operations, parts shown, interactions, grouping
- Level of detail (30%): a design, or a map of files?
- Readability: arrows through boxes, overlapping labels, crossings
- Score = content × (0.5 + 0.5 × readability). Content matters most, and a hard-to-read diagram loses up to half its score.
Each model gets three attempts and the score is the average.
Testing the judge before trusting it
A judge model is only useful if it notices what's wrong, so I tested the judge before scoring any model. I made copies of my reference diagram, each damaged in one known way:
- no starting point
- no database
- three agents merged into one
- no grouping
- a component that doesn't exist in the code
The judge has to mark each damaged copy below the clean reference, on the mark that damage should affect.
The first version of the judge missed a defect. Giving it the same file tools the models have, plus a one-time study of the repository, fixed that: it then caught all five. The benchmark ships with this check as its own task, so anyone can rerun it when they swap the judge.
Models Tested
| Model | Why |
|---|---|
| Claude Sonnet 5.5 | I usually use Opus, so I wanted to see whether Sonnet gives comparable results |
| Claude Opus 5.5 | The one I use most. Does the most capable model actually draw the best diagram? |
| GPT-5.5 | The same question for OpenAI's large model |
| GPT-5.4 mini | A small, cheap model, to test the price trade-off |
| Gemini 3.8 Flash | I rarely use Gemini and wanted to see how it compares |
| Gemini 3.1 Pro | Google's larger model. It is also the judge, so read its score with that in mind |
I normally reach for Claude and OpenAI models, so I wanted to see how Gemini compares. I also wanted to know whether the more expensive models are worth their tokens for an everyday task like this, so the lineup has a large and a small model from each provider where I could get one.
Findings
| Model | Average score | Model cost per attempt |
|---|---|---|
| Claude Sonnet 5.5 | 0.44 | $0.19 |
| Claude Opus 5.5 | 0.44 | $0.44 |
| GPT-5.5 | 0.43 | $0.39 |
| Gemini 3.1 Pro (the judge) | 0.43 | $0.25 |
| Gemini 3.8 Flash | 0.43 | $0.22 |
| GPT-5.4 mini | 0.38 | $0.04 |
My reference scores 1.0. The best single attempt, from Claude Opus 5.5, scored 0.56. On average, Sonnet and Opus scored the same.
These numbers are from my own runs of the task in the benchmark notebook. The Kaggle leaderboard runs the same task again for each model, so its scores will differ a little from run to run, and it may list fewer models while my quota catches up.
1. You don't need the expensive model for this
This was the result I cared about most. Claude Opus 5.5 costs eleven times as much per attempt as GPT-5.4 mini, and the difference in score is 0.06, which is smaller than the variation between one model's own attempts. Sonnet matched Opus at less than half the price.
For drawing an architecture diagram from a repository, the small models held their own. The assumption that a larger model will produce a better result doesn’t always hold, and that was something I learned along the way.
2. There is a lot of headroom
Every model produced a valid, renderable draw.io diagram with a clear structure, which is a real achievement. But none came close to the reference. My sketch has about ten boxes; the models drew several times that, most with file names inside, and level of detail was marked at half or lower on every attempt.
The clearest example: BillTalk's design is three voice agents, each with its own system prompt and logic, sharing one on-device model. In the diagrams I examined, the models didn't draw three agents. They drew three UI files, or one box called "system prompts". The benchmark has plenty of room for future models to improve.
3. Models describe the code, not the system
This is the insight I didn't expect, and it came from looking at where the diagrams and my sketch disagreed.
"Component." My first prompt said "show every component". I meant an architectural building block: the database, an agent, the speech recogniser. But BillTalk is a React Native app with a folder called components/, where the word means a piece of UI. The early diagrams had boxes for the files in that folder, down to the save button.
"Belong together." My first prompt also asked to group components that "belong together". I grouped by where things run: everything on the user's device in one group. The early diagrams grouped by folder: UI layer, state layer, data layer.
So I reworded both lines to what you see above: "every major part of the system", and groups defined by where things run. All the scores in this post are from that reworded prompt.
Fixing the words wasn't enough. Every model still scored half or lower on level of detail, on every attempt. So the over-detail isn't a misunderstanding of a word. Given a repository, these models default to describing how the code is organised, and "high-level" on its own doesn't pull them away from that.
A person reads "high-level design" and draws a dozen boxes. If you want that from an LLM today, be explicit: say roughly how many boxes you expect, give it a rough sketch or a list of the parts you want shown, and say that groups should reflect where things run, not which folder they're in.
4. What building the scoring taught me
I looked at every picture next to its score, and that shaped the scoring more than anything else:
- Readability should scale the score, not add to it. My first version added the parts together, and good content covered for arrows running through boxes. Multiplying fixed that.
- A multiplier needs a floor. Scaling by readability alone squashed every model into the same narrow band. Letting readability remove at most half gave scores that matched my own ranking of the diagrams.
- Judges anchor on numbers you give them. A limit I wrote into the instructions ("at most 2") became the judge's default answer. Describing what to look for works better than stating limits.
- Make the judge list what it sees first. Asking for the specific defects before the mark produced reviews I could check against the picture, like "the arrow from Voice command passes through the Shop state cache box".
The scoring I ended with agreed with my own judgement on the diagrams I compared by eye, including which one was best.
Upcoming
This is version one. Here's what I'm building next:
- Finer readability scoring. The judge currently gives most diagrams the same readability mark. Counting defects per arrow will separate a slightly messy diagram from a tangled one, and should spread the models further apart.
- An explicit box budget. Does telling the model "about ten boxes" produce a usable design, or just a shortened file map?
- More repositories. BillTalk runs entirely on a phone. Next is a web app with a server, a hosted database and an external AI service, so the benchmark covers both kinds of architecture.
- A second task: workflow diagrams from a document. Give the model a design doc or paper and ask for one specific workflow, to test turning prose into a diagram.
- Human ratings. Rank a set of diagrams by hand and tune the scoring against people.
- More attempts per model, to tighten the averages.
My Benchmark
All the benchmark code is in the notebook, in plain cells you can read top to bottom. The dataset holds only the BillTalk repository and my reference diagram.
Built with the kaggle-benchmarks SDK. Diagrams are rendered with the draw.io viewer through Playwright.
Top comments (0)