Can Modern VLMs Actually Reason Through Spatial Grid Puzzles?

Fahmi Agung Maulana — Sun, 11 Oct 2026 23:14:45 +0000

What I Benchmarked

Current Frontier Visual Language Models (VLMs) excel at multi-modal reasoning and captioning, but how well do they execute step-by-step visual spatial verification?

To test this capability, I created VCIS-Bench (Visual Constraint & Instruction Simulation), a synthetic 7×7 grid puzzle benchmark. Each puzzle features a strict layout: an Agent (A), Key (K), Locked Door (D), Goal (G), and Walls (#). The agent must follow strict cardinal mechanics—it cannot pass through walls, and it must collect the key on the left side of a central partition before unlocking the door to reach the goal on the right.

I evaluated the models across 3 core tasks:

Task 1: Overall Plan Verification: Given an action sequence, determine whether the proposal is VALID or INVALID.
Task 2: Spatial Grounding Verification: Evaluate spatial awareness by verifying object coordinates relative to labeled row (R1–R7) and column (C1–C7) headers.
Task 3: Step-Level Error Diagnosis: Identify the exact 1-based step index ($1, 2, 3, \dots$) where an action plan first violates a rule (e.g., hitting a wall or passing through a door without a key).

Models Tested

I evaluated three distinct models across different parameter tiers to measure accuracy, latency, and cost-efficiency:

Claude Sonnet 5.5 (Anthropic): Benchmark state-of-the-art vision reasoning.
Gemini 3.7 Flash (Google): Google’s fast multi-modal flash model.
Gemma 4 26B A4B (Google): Open-weights vision-language architecture.

These models provide a clear perspective on the performance gap between top-tier proprietary frontier models, lightweight API endpoints, and open-weights models.

Findings

Here is how the models performed across the benchmark suite:

Leaderboard Overview

Task	Claude Sonnet 5.5	Gemini 3.7 Flash	Gemma 4 26B A4B
Task 1: Overall Plan Verification	100.0%	100.0%	83.3%
Task 2: Spatial Grounding Verification	100.0%	100.0%	96.7%
Task 3: Step-Level Error Diagnosis	100.0%	92.0%	76.0%
Overall Score	1.00	0.97	0.85

(Data evaluated on October 11, 2026)

Key Takeaways

Step-Level Simulation is the Hardest Bottleneck: While Claude Sonnet 5.5 achieved a perfect 100% accuracy across all tasks, performance dropped for smaller models on Task 3. Gemini 3.7 Flash reached 100% on binary verification tasks (Tasks 1 and 2), but dropped to 92.0% when required to pinpoint the exact 1-based step index of a failure.
Spatial Coordination vs. Open-Weight Execution: Gemma 4 26B A4B performed strongly on Task 2 spatial coordinate checks (96.7%), proving that open-weight models are proficient at vision grounding. However, its performance decreased to 76.0% on step-level execution (Task 3) due to tracking errors across multi-step state transitions.
Cost-Efficiency Sweet Spot: Looking at the Score vs. Total Cost Pareto Chart, Gemini 3.7 Flash offered a favorable efficiency tradeoff—delivering a near-perfect 0.97 aggregate score at roughly half the total cost and execution latency of Claude Sonnet 5.5.

My Benchmark

Check out the task code and interactive leaderboard runs on Kaggle Benchmarks:

📌 VCIS Benchmark Overview & Leaderboard: https://www.kaggle.com/benchmarks/fahmiagungmaulana/vcis
🧩 VCIS Task 1 (Overall Plan Verification): https://www.kaggle.com/benchmarks/tasks/fahmiagungmaulana/vcis-task-1-overall-plan-verification/1
📐 VCIS Task 2 (Spatial Grounding Verification): https://www.kaggle.com/benchmarks/tasks/fahmiagungmaulana/vcis-task-2-spatial-grounding-verification/3
🛠️ VCIS Task 3 (Step-Level Error Diagnosis): https://www.kaggle.com/benchmarks/tasks/fahmiagungmaulana/vcis-task-3-step-level-error-diagnosis/1