Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.
Large language models (LLMs) can draft Python scripts, pass medical board exams, and write poetry. But can they actually visualize the physical world, or are they simply echoing text patterns from their training data?
Standard coding benchmarks like HumanEval or MBPP test syntactic correctness and procedural logic. However, they tell us very little about an AI’s internal spatial awareness. When an LLM claims to understand concepts like "next to," "overlapping," or "below," is it truly reasoning about visual space, or is it just predicting token sequences?
To answer this, we need a test that bridges text generation and visual reality.
The Problem: The Visual Blindspot in LLM Benchmarks
Traditional text-based code evaluation suffers from a major visual blindspot:
- Syntax ≠ Spatial Understanding: A model can produce syntactically valid Python or HTML while remaining completely blind to the visual outcome.
- Pixel Matching Is Too Rigid: Evaluating image generation models using metrics like FID (Fréchet Inception Distance) or exact pixel matching fails for vector graphics. A red circle shifted 2 pixels to the left isn't a failure, but rigid pixel-based evaluation penalizes it as one.
- Multimodal Blindspots: Vision-Language Models (VLMs) can describe an image, but asking a model to build a scene from scratch requires true 2D spatial arrangement, layering order, perspective, and domain knowledge.
Without a way to evaluate how code renders into visual spatial logic, we cannot accurately measure an AI's mental model of the physical world.
Objectives of SVGBench
SVGBench was built to turn raw vector code into a quantitative benchmark for spatial reasoning. Its primary goals include:
- Test Visual Spatial Logic: Measure whether models understand anatomical placement, relative positioning, and physical interactions (e.g., placing a saddle on a horse, or floating an object in water).
- Evaluate Precision Instruction Following: Force models to satisfy complex prompts containing 10–15 distinct visual sub-requirements simultaneously.
- Establish Objective Multimodal Scoring: Move away from binary pass/fail syntax tests or fuzzy global image scores by using requirement-based evaluation.
- Provide an End-to-End Automated Pipeline: Offer an automated workflow that generates SVG code, renders it via headless browsers, and evaluates the visual output.
The Conceptual Solution: Code, Render, Judge
Instead of asking a model to describe a scene, SVGBench forces the model to express the scene programmatically using Scalable Vector Graphics (.svg).
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Prompt & │ ──> │ Target Model │ ──> │ Headless │ ──> │ Evaluator │
│ Requirements │ │ Generates │ │ Browser │ │ Vision LLM │
│ (JSON) │ │ SVG Code │ │ Renders PNG │ │ Scores Match │
└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
Conceptually, the evaluation works in three distinct steps:
- The Code Phase: The target model receives a prompt along with detailed requirements (e.g., "Draw a toy rocket with a blue body, red nose cone, and three symmetrical base fins"). It must output valid SVG code.
- The Render Phase: The raw SVG string is rendered into a high-resolution PNG image inside an isolated headless Chrome browser. This turns abstract code into actual visual pixels.
- The Judgment Phase: A multimodal Evaluator LLM (e.g., GPT-4o) inspects both the rendered PNG image and the raw SVG source code against every single requirement. The final score represents the percentage of visual and structural requirements successfully met ($0.0$ to $1.0$).
Strengths & Weaknesses
Like any benchmark architecture, SVGBench makes intentional trade-offs.
Strengths
- Granular Evaluation: Scoring individual requirements produces a nuanced percentage score rather than a misleading pass/fail metric.
-
Resilient to Formatting Differences: SVG allows infinite ways to draw the same object. By evaluating rendered images alongside code, models aren't penalized for using different drawing techniques (e.g.,
<path>vs.<rect>). - Tests Multimodal Synthesis: It evaluates code synthesis, visual aesthetic arrangement, coordinate math, and domain knowledge simultaneously.
Weaknesses
- Judge LLM Dependency: Evaluation quality depends on the vision model's ability to accurately spot fine visual details in the rendered PNG.
- 2D Vector Limitation: SVGBench evaluates 2D layouts and pseudo-3D projections; it does not measure native 3D mesh or spatial depth generation.
- Execution Overhead: Running headless browser screenshots and calling multimodal evaluator APIs makes evaluation slower and more resource-intensive than running standard unit tests.
Under the Hood: System Architecture & Execution
Let's look at how SVGBench handles this pipeline under the hood.
SVGBench Architecture
├── questions/
│ ├── questions.json # Main benchmark (100+ complex prompts)
│ └── test_questions.json # 3-question smoke test dataset
├── src/
│ ├── run.py # Main CLI controller
│ ├── benchmark/
│ │ └── benchmark.py # Multi-threaded execution engine
│ └── utils/
│ ├── llm.py # API client with vision support
│ └── svg_renderer.py # Selenium & Headless Chrome renderer
└── results/ # Generated artifacts (.svg, .png, JSON)
1. Headless Browser Rendering (src/utils/svg_renderer.py)
To render SVG code accurately, SVGRenderer uses Selenium with Google Chrome:
- Calculates bounding boxes if explicit
viewBoxdimensions are missing. - Wraps raw SVG code into an inline HTML5 document.
- Captures a high-resolution screenshot to convert the vector graphics into a PNG raster image.
2. Parallel Benchmark Engine (src/benchmark/benchmark.py)
To process datasets efficiently, Benchmark uses Python's ThreadPoolExecutor to evaluate multiple questions in parallel. It saves progress automatically, allowing you to resume interrupted benchmark runs without re-evaluating completed items.
3. Quick Testing vs. Full Evaluation
SVGBench includes two test sets:
-
questions/questions.json: The standard dataset containing over 100 complex prompts (covering UI layouts, physical scenes, and spatial positioning). -
questions/test_questions.json: A lightweight 3-question dataset for sanity-checking your setup without burning API credits.
How to Run SVGBench
Setup
# Clone and prepare the environment
git clone https://github.com/johnbean393/SVGBench.git
cd SVGBench
conda create -n svgbench python=3.13
conda activate svgbench
pip install -r requirements.txt
Note: Ensure Google Chrome and chromedriver are installed and added to your system PATH.
Execution
Configure your .env file with your API key:
OPENROUTER_API_KEY=your_key_here
Run the evaluation across one or multiple models:
# Evaluate a single model
python src/run.py --model "anthropic/claude-sonnet-4"
# Benchmark multiple models sequentially
python src/run.py --model "google/gemini-2.5-pro;anthropic/claude-sonnet-4;openai/gpt-4.1"
After the benchmark completes, launch the built-in web UI to inspect the results:
python -m http.server 8000 --directory results
Leaderboard Results
Here is how leading AI models perform on SVGBench:
| Rank | Model | Score |
|---|---|---|
| 1 | Claude Opus 4.6 (Medium) | 75.6% |
| 2 | GPT-5.2 (X-High) | 74.4% |
| 3 | Claude Opus 4.5 | 72.0% |
| 4 | GLM-5 | 70.3% |
| 5 | Gemini 3 Pro Preview | 68.7% |
| 6 | Kimi K2.5 | 67.5% |
| 7 | GPT-5.1 (High) | 67.5% |
| 8 | MiniMax M2.5 | 65.0% |
| 9 | Claude Sonnet 4.5 | 62.2% |
| 10 | Gemini 2.5 Pro | 61.4% |
| 11 | o3 | 56.7% |
| 12 | DeepSeek V3.1 | 53.1% |
(Source: SVGBench GitHub Repository)
Key Takeaways
- Frontier Models Lead Visual Reasoning: Top reasoning and frontier models like Claude Opus 4.6 and GPT-5.2 score around 75%, demonstrating strong spatial composition skills.
- Significant Room for Improvement: Even top-tier models miss roughly 25% of granular visual instructions, often struggling with precise overlapping elements, complex alignment, or coordinate math.
- Code Intelligence ≠ Visual Intelligence: Strong text coding performance does not always translate to high visual fidelity, proving that spatial vector reasoning requires unique capabilities beyond standard code generation.
By combining raw code generation with automated visual evaluation, SVGBench provides a clear window into how well AI models truly understand the visual and spatial world.
*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*
Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.
HexmosTech
/
git-lrc
Free, Micro AI Code Reviews That Run on Git Commit
| 🇩🇰 Dansk | 🇪🇸 Español | 🇮🇷 Farsi | 🇫🇮 Suomi | 🇯🇵 日本語 | 🇳🇴 Norsk | 🇵🇹 Português | 🇷🇺 Русский | 🇦🇱 Shqip | 🇨🇳 中文 | 🇮🇳 हिन्दी |
git-lrc
Free, Micro AI Code Reviews That Run on Commit
GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.
git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.
In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen
At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…
Top comments (0)