Can Modern VLMs Actually Reason Through Spatial Grid Puzzles?

What I Benchmarked

Current Frontier Visual Language Models (VLMs) excel at multi-modal reasoning and captioning, but how well do they execute step-by-step visual spatial verification?

To test this capability, I created VCIS-Bench (Visual Constraint & Instruction Simulation), a synthetic 7×7 grid puzzle benchmark. Each puzzle features a strict layout: an Agent (A), Key (K), Locked Door (D), Goal (G), and Walls (#). The agent must follow strict cardinal mechanics—it cannot pass through walls, and it must collect the key on the left side of a central partition before unlocking the door to reach the goal on the right.

I evaluated the models across 3 core tasks:

1. Task 1: Overall Plan Verification: Given an action sequence, determine whether the proposal is VALID or INVALID.
2. Task 2: Spatial Grounding Verification: Evaluate spatial awareness by verifying object coordinates relative to labeled row (R1–R7) and column (C1–C7) headers.
3. Task 3: Step-Level Error Diagnosis: Identify the exact 1-based step index ($1, 2, 3, \dots$) where an action plan first violates a rule (e.g., hitting a wall or passing through a door without a key).

---

Models Tested

I evaluated three distinct models across different parameter tiers to measure accuracy, latency, and cost-efficiency:

• Claude Sonnet 5.5 (Anthropic): Benchmark state-of-the-art vision reasoning.
• Gemini 3.7 Flash (Google): Google’s fast multi-modal flash model.
• Gemma 4 26B A4B (Google): Open-weights vision-language architecture.

These models provide a clear perspective on the performance gap between top-tier proprietary frontier models, lightweight API endpoints, and open-weights models.

---

Findings

Here is how the models performed across the benchmark suite:

Leaderboard Overview

Task	Claude Sonnet 5.5	Gemini 3.7 Flash	Gemma 4 26B A4B
Task 1: Overall Plan Verification	100.0%	100.0%	83.3%
Task 2: Spatial Grounding Verification	100.0%	100.0%	96.7%
Task 3: Step-Level Error Diagnosis	100.0%	92.0%	76.0%
Overall Score	1.00	0.97	0.85

(Data evaluated on October 11, 2026)

---

Key Takeaways

• Step-Level Simulation is the Hardest Bottleneck: While Claude Sonnet 5.5 achieved a perfect 100% accuracy across all tasks, performance dropped for smaller models on Task 3. Gemini 3.7 Flash reached 100% on binary verification tasks (Tasks 1 and 2), but dropped to 92.0% when required to pinpoint the exact 1-based step index of a failure.
• Spatial Coordination vs. Open-Weight Execution: Gemma 4 26B A4B performed strongly on Task 2 spatial coordinate checks (96.7%), proving that open-weight models are proficient at vision grounding. However, its performance decreased to 76.0% on step-level execution (Task 3) due to tracking errors across multi-step state transitions.
• Cost-Efficiency Sweet Spot: Looking at the Score vs. Total Cost Pareto Chart, Gemini 3.7 Flash offered a favorable efficiency tradeoff—delivering a near-perfect 0.97 aggregate score at roughly half the total cost and execution latency of Claude Sonnet 5.5.

---

My Benchmark

Check out the task code and interactive leaderboard runs on Kaggle Benchmarks:

• 📌 VCIS Benchmark Overview & Leaderboard: https://www.kaggle.com/benchmarks/fahmiagungmaulana/vcis
• 🧩 VCIS Task 1 (Overall Plan Verification): https://www.kaggle.com/benchmarks/tasks/fahmiagungmaulana/vcis-task-1-overall-plan-verification/1
• 📐 VCIS Task 2 (Spatial Grounding Verification): https://www.kaggle.com/benchmarks/tasks/fahmiagungmaulana/vcis-task-2-spatial-grounding-verification/3
• 🛠️ VCIS Task 3 (Step-Level Error Diagnosis): https://www.kaggle.com/benchmarks/tasks/fahmiagungmaulana/vcis-task-3-step-level-error-diagnosis/1