Jirawat Boonkumnerd

Posted on Oct 23

ZeroVisual: I Was Too Lazy to Do Manual UI Testing, So I Made Open-Source AI Do It

#devchallenge #hf26challenge #opensource #ai

Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady Submission 🦥

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

I am lazy to do manual UI testing. Truly, unapologetically lazy.

There is nothing quite as soul-crushing on a Friday afternoon as manually resizing browser windows, squinting at full-HD screenshots to see which button shifted by 3 pixels, inspecting CSS flexbox margins on three different viewports, or calculating whether a muted gray button passes WCAG 2.1 AA color contrast.

When Vision LLMs arrived, the promise seemed magical: "Just dump a screenshot into an AI agent and let it review your UI!"

Except laziness quickly met reality:

The Vision Token Explosion: Feeding a single raw 1920×1080 screenshot to a Vision Transformer (ViT) burns 1,000 to 1,600 image tokens. Multiply that across Desktop, Tablet, and Mobile across a 30-test suite, and your CI/CD pipeline burns through hundreds of thousands of tokens per commit.
The 5MB Base64 Payload Pipe Choke: Shipping multi-megabyte Base64 image payloads over JSON-RPC or stdio to your local coding agent creates noticeable lag and IPC stalls.
Canvas Noise Hallucinations: When an LLM looks at an entire webpage canvas, it gets easily distracted by navigation links, avatars, and footer copyrights, often hallucinating defects on pixels that never actually changed.

Sloths have it figured out: move at your own pace, conserve energy, and let the work do itself. So in honor of International Sloth Day, I built ZeroVisual—an open-source, zero-API-key visual regression and automated UI design QA pre-processor that acts as a token-optimizing firewall between browser tests and Vision LLMs.

What I Built

ZeroVisual is a local pre-processing engine, custom Playwright Reporter, CLI, and Model Context Protocol (MCP) server. It cuts Vision LLM token usage by up to 75% on desktop viewports and network payloads by up to 98%, without requiring a single external API key.

It addresses the two most tedious UI tasks developers dread:

Automated Visual Regression (zerovisual regression & Playwright Custom Reporter):
Instead of immediately bombarding a vision model with full-page screenshots, ZeroVisual relies on a Two-Stage Architecture:
- Stage 1 (0 Tokens): Deterministic pixel diffing runs locally for free. If all visual tests pass, 0 AI tokens are consumed and CI completes in milliseconds.
- Stage 2 (Optimized Triage): When a test fails, ZeroVisual intercepts the Playwright diff, isolates the failure bounding box (with 32px contextual padding), resizes the crop to match the model's exact Vision Transformer patch grid, scopes the Accessibility Object Model (ariaSnapshot), and aggregates Desktop, Tablet, and Mobile failures under one testId. Your AI agent diagnoses the responsive layout bug in a single prompt.
Automated Design QA (zerovisual design-qa):
For developers building web apps without a dedicated UI/UX designer, guessing spacing and accessibility in CSS is exhausting. zerovisual design-qa audits any localhost URL against mathematical heuristics (8pt spatial grid rhythm, WCAG 2.1 contrast math, 44×44px touch targets) and compares live DOM against open-source Penpot design exports—zero proprietary Figma lock-in.
Built-in MCP Server & Agent Skills:
ZeroVisual exposes native tools over stdio (list_visual_failures, get_visual_failure_bundle, audit_live_ui_design, optimize_image) and includes strictly-typed Agent Skills (zerovisual-regression, zerovisual-design-qa, zerovisual-optimize) compatible with Cursor, Claude Code, Antigravity, and custom CI agents.

[Stage 1: Deterministic Engine (0 Tokens)]
Playwright Pixel Diff / CSS Heuristic Math / Penpot Token Diff
       │
       ├──► PASS ──► Workflow Ends Immediately (0 AI Tokens Used)
       │
       └──► FAIL / AUDIT ──► Triggers ZeroVisual Pre-Processor
                                    │
[Stage 2: ZeroVisual Optimizer]     ▼
  1. Detects diff bounding box + 32px context padding
  2. Resizes & encodes to WebP based on Model Tile Grid (512px / 504px / 768px / 448px)
  3. Scopes Accessibility Tree (ariaSnapshot) & 14 Computed CSS layout styles
  4. Aggregates Desktop + Tablet + Mobile viewports by testId
                                    │
                                    ▼
              Outputs ./test-results/triage-ready.json
                                    │
            ┌───────────────────────┴───────────────────────┐
            ▼                                               ▼
   [Local IDE via MCP]                             [CI/CD Pipeline CLI]
Cursor / Claude Code / Antigravity               GitHub Actions / GitLab CI
queries ZeroVisual MCP Server + Skill          reads JSON & posts PR Verdict

Demo

🌐 Live Documentation & Interactive Benchmark Explorer: zerovisual.hotcode.dev/guides/benchmarks/
🧮 Interactive Token Calculator: Calculate exact ViT token and byte savings directly on the homepage for OpenAI, Anthropic, Gemini, Gemma, and Local models.
🔬 Real-World Benchmark Explorer: Inspect side-by-side screenshots (Baseline, Actual, Diff, Crop) across live production websites (Wikipedia, Hacker News, GitHub, MDN) and examine actual local VLM reasoning traces (<think>).

Quick Start in Playwright

Add ZeroVisual as a custom reporter in playwright.config.ts:

// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  projects: [
    { name: 'Desktop Chrome', use: { ...devices['Desktop Chrome'] } },
    { name: 'Mobile Safari', use: { ...devices['iPhone 14'] } },
  ],
  reporter: [
    ['list'],
    ['zerovisual/reporter', {
      targetProvider: 'gemma', // 'gemma' | 'gemini' | 'openai' | 'anthropic' | 'qwen' | 'local'
      padding: 32,             // Context padding around diff bounding box
      outputDir: './test-results'
    }]
  ],
});

Run Standalone Design QA on Localhost

# Audit live local UI against 8pt grid, WCAG contrast, and tap targets
npx zerovisual design-qa --url http://localhost:3000 --provider gemma

# Or compare directly with an open-source Penpot design artboard
npx zerovisual design-qa --url http://localhost:3000 --penpot ./designs/artboard.svg

Connect to Coding Agents via MCP

# Register with Claude Code
claude mcp add zerovisual -- npx -y zerovisual-mcp

# Or install Agent Skills for Antigravity / Cursor
npx skills add hotcode-dev/zerovisual

Code

hotcode-dev / zerovisual

Zero-waste, zero-API-key visual regression and automated UI design QA pre-processor for AI Agents.

ZeroVisual

Zero-waste, zero-API-key visual regression and automated UI design QA pre-processor for AI Agents.

ZeroVisual is a local pre-processing engine, Playwright Custom Reporter, CLI, and Model Context Protocol (MCP) server. It acts as a token-optimizing firewall between your browser tests/UI and your Vision LLMs—reducing vision token consumption by up to 90% without requiring a single LLM API key.

🌐 Full Documentation & Interactive Token Calculator: Visit our Astro Documentation Site. Local dev: npm run docs:dev.

🎯 What Is This For?

ZeroVisual solves two major pain points in modern software engineering:

1. For QA & Full-Stack Engineers: Automated Visual Regression (`regression` mode)

Standard Playwright toHaveScreenshot() pixel-diffing is fast and free, but fails on harmless sub-pixel font anti-aliasing or minor OS rendering shifts. Conversely, sending raw 1920x1080 screenshots across Desktop, Tablet, and Mobile viewports to a Vision LLM to triage failures will burn through your token budget and slow…

View on GitHub

The repository is 100% open source under the Apache-2.0 License:

Source Code: github.com/hotcode-dev/zerovisual
Tech Stack: TypeScript, @modelcontextprotocol/sdk, @playwright/test, sharp, citty, zod, fast-xml-parser, Astro + Starlight.

How I Built It

ZeroVisual is designed from the ground up around open-source AI and local inference:

Local Vision-Language Models (Open-Weight ViTs): We benchmarked and validated ZeroVisual against Google's google/gemma-4-26B-A4B-it multimodal MoE architecture running completely locally on our workstation via vLLM. Because open-weight vision models (such as Gemma 4, Qwen 2.5/3.8, and PaliGemma) publish their exact Vision Transformer patch dimensions (e.g. 28×28px or 14×14px patch grids), ZeroVisual mathematically fits cropped diffs into the exact minimum patch allocation:
- Google Gemma / Gemini (gemma / gemini): 768×768px native tile (or 384×384px in lowTokenMode)
- Anthropic / Qwen grid (anthropic / qwen): 504×504px (exact 18×18 grid of 28×28px patches)
- OpenAI (openai): 512×512px (single tile)
- Local VLMs (local): 448×448px default (or custom ViT patch dimensions)

Model Vision Grid Reference

Provider Setting	Target Max Bounding Box	Vision Grid Mechanics	Approx. Token Cost per Diff
`openai`	`512 × 512 px`	Fits inside 1 standard OpenAI `512px` tile	~85 – 255 tokens
`anthropic`	`504 × 504 px`	Exact `18 × 18` grid of `28×28 px` patches	~28 – 324 tokens
`gemma` / `gemini`	`768 × 768 px`	Fits 1 native Gemma / Gemini multimodal tile	~258 tokens
`gemma` (`lowTokenMode`)	`384 × 384 px`	Sub-384px single base tile optimization	~258 tokens (ultra-fast)
`qwen`	`504 × 504 px`	Dynamic 2D spatial patch grid (`28×28 px`)	~28 – 324 tokens
`local`	`448 × 448 px` (configurable)	Optimized for local VLMs (`28px` / `14px` patch ViTs)	Local compute

Model Context Protocol (MCP) Standard:
Built on @modelcontextprotocol/sdk, exposing standard tools and prompts over stdio. Any MCP-compatible agent harness (Claude Code, Cursor, Windsurf, Antigravity) can query test results, inspect failure bundles, and trigger UI audits without custom glue code.
Deterministic Local Math Pre-Engine:
- Bounding Box Math: Fast pixel difference scan using sharp buffer analysis with multi-color palette support (Playwright magenta #ff00ff, rose, and red).
- Accessibility Scoping: Extracts Playwright's ariaSnapshot YAML only for the failing element subtree, pairing visual evidence with clean semantic hierarchy.
- CSS Layout Heuristics: Computes WCAG 2.1 contrast ratios and 8pt spatial grid alignment on the live DOM before any AI inference takes place.
Isolated Agent Skills:
Ships strictly-typed Agent Skills (zerovisual-regression, zerovisual-design-qa, zerovisual-optimize) that instruct coding models with precise JSON schemas and design rules, preventing instruction dilution.

Why Does Open Innovation Matter?

Open innovation is what makes ZeroVisual fundamentally possible. A closed API approach wouldn't work for three critical reasons:

Privacy & Zero Data Leaks:
UI test suites run against staging dashboards, internal admin portals, checkout flows, and unreleased features. Uploading full-resolution screenshots of private company data to commercial cloud APIs violates enterprise security and compliance policies. With open-weight models (Gemma, Qwen) running locally through ZeroVisual, zero pixels ever leave your local machine or internal CI runner.
ViT Patch Mechanics Transparency:
Proprietary vision APIs treat image processing as an opaque black box. In contrast, open-source model architectures expose their exact Vision Transformer tokenization logic (patch sizes, position embeddings, and aspect ratio bucketing). This open transparency allowed us to engineer mathematically optimal cropping algorithms that align pixel boundaries directly with the model's ViT patches.
Uncapped, Meter-Free Automation:
Commercial vision APIs charge between $0.01 and $0.03 per image. In an active development team running hundreds of Playwright visual tests across multiple responsive viewports, running visual AI triage on every PR would incur crippling costs. Open innovation and local inference liberate developers from API meter anxiety—allowing you to run thorough UI testing on every single commit for $0.

My Agent Session

Judges and readers can inspect the full interactive agent session trace showing the benchmark execution and triage optimization on local open-weight vision models:

ZeroVisual Vision LLM Empirical Benchmark: 87.7% ViT Token Reduction on Local Qwen-27B RadixArk/Qwen3.8-27B-NVFP4

You

Run empirical benchmark comparing Vision LLM visual regression triage with and without ZeroVisual on local Qwen 27B model (RadixArk/Qwen3.8-27B-NVFP4 via SGLang). Measure token consumption, latency, and accuracy across 6 realistic web-UI test cases.

Agent

Executed empirical evaluation across 6 real-world web UI visual regression scenarios:

Wikipedia Portal Search Offset (Layout Shift)
GitHub PR Merge Box Regression (Button Sizing/Shift)
SaaS Pricing Card Contrast Violation (WCAG 2.1 AA)
E-Commerce Checkout Collision (Flexbox Overlap)
Dashboard Stat Overflow & Clipping (Typography/Height)
Mobile Safari Nav Drawer Overflow (Breakpoint/Z-Index)

Empirical Results Summary:
- Total Tokens Consumed: 11,556 tokens (Raw) vs 5,421 tokens (ZeroVisual) -> 53.1% reduction
- Vision Image Tokens (ViT): 5,252 tokens (Raw) vs 644 tokens (ZeroVisual) -> 87.7% reduction
- Network Bandwidth: 146.6 KB vs 21.7 KB -> 85.2% reduction
- Average Latency: 53.3s vs 36.4s -> 32% faster (up to 3.5x faster on complex viewports)
- Average Triage Accuracy: 76.7% vs 82.5% -> +5.8% accuracy (zero canvas noise hallucinations)

Core Engine Improvements Implemented:
- Added 'qwen' provider support with native 28x28 patch grid dimension math (504x504 max tile).
- Upgraded defaultDiffPixelMatcher to support Playwright magenta (#ff00ff), rose, and high-saturation pixelmatch reds across arbitrary dark/light themes.
- Added automated benchmark CLI runner and testdata generators.

Direct session link: View on DEV Agent Sessions

The Lazy Payoff

How much effort does ZeroVisual actually save?

We ran an empirical benchmark across 11 visual regression scenarios—including 5 live production websites (Wikipedia Desktop & Mobile, Hacker News, GitHub Pricing, MDN Web Docs) and 6 synthetic component regressions—comparing raw viewport screenshots against ZeroVisual optimization on local Gemma 4 26B (google/gemma-4-26B-A4B-it served via vLLM):

Core Metric	Without ZeroVisual (Raw Viewport)	With ZeroVisual (Optimized)	The Lazy Payoff
Vision Image Tokens (ViT)	9,030 tokens	2,838 tokens	-68.6% ViT Token Reduction
Network Payload Transfer	638.9 KB	116.2 KB	-81.8% Bandwidth Reduced
Total Tokens Consumed	7,612 tokens	7,148 tokens	-6.1% Total Tokens Saved
Average Roundtrip Latency	8.4 seconds	7.3 seconds	13% Faster (up to 37% on desktop)
Average Triage Accuracy	59.1%	65.5%	+6.4% Accuracy Boost (Zero Canvas Noise)

Real-World Highlights: Live Production Sites

Wikipedia Desktop (1280×800): Payload shrank from 122.3 KB down to 11.1 KB (-91.0%), and roundtrip triage latency dropped from 11.3s down to 7.2s (37% faster).
Wikipedia Mobile (390×664): The raw viewport scored 0% accuracy due to full canvas noise. Cropping to the 370×133 failure bounding box (1.5 KB WebP, -96.9% payload) boosted diagnostic accuracy to 35%.
MDN Web Docs Mobile (390×664): Diagnosing a subtle search input contrast defect jumped from 30% accuracy up to 65% accuracy (+35% boost) while cutting network payload from 55.1 KB down to 4.7 KB (-91.5%).

The Real Payoff: Instead of spending 20 minutes manually reproducing a broken responsive layout and inspecting DevTools styles, ZeroVisual intercepts the failure, hands a token-optimized bundle to the local AI agent, and outputs the exact CSS diff to paste into your pull request.

Prize Categories

Overall Winner: Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady (Rewarding Laziness with Local Open-Source AI)
Best Use of Gemma: Evaluated and benchmarked ZeroVisual against Google's google/gemma-4-26B-A4B-it multimodal MoE model running locally via vLLM.

Top comments (2)

Alex @Bickov • Oct 25

The pixel diffing stage is the right call, most token waste comes from sending full page screenshots that never needed a model at all. One number worth having on hand for the triage stage: current Claude models cap images at 2576px on the long edge, older ones at 1568px, so whatever you crop and resize for the failure case should target the actual model ceiling, not a generic max size. I build SlimSnap, a free screenshot tool, and ran into the same tile grid mismatch when resizing for Claude versus GPT vision models, worth checking both if ZeroVisual ever needs to target more than one backend.

Jirawat Boonkumnerd • Oct 26

Cool, ZeroVisual has the same idea of token saving as Slimsnap; it doesn't have a screenshot part, just the preprocessing of the image before sending it to the LLM.