I Built an AI That Reads Your Screen and Auto-Fills Forms — Here's How

#ai #automation #productivity #showdev

Every day, millions of knowledge workers do the same thing: look at data on one screen, then manually type it into another application. Invoice numbers, tracking codes, customer details, form fields — copied by hand, one keystroke at a time.

I got tired of it. So I built Scan0tron — a desktop AI that literally sees your screen and auto-fills forms for you.

The Problem Nobody Talks About

We have AI that writes essays, generates images, and passes the bar exam. But somehow we're still manually copying data between windows like it's 1998.

Think about how much time you spend on:

Reading an invoice PDF → typing values into accounting software
Copying tracking numbers from email → pasting into shipping portals
Transcribing handwritten notes → entering into a CRM
Reading one spreadsheet → filling out fields in another app

This isn't "work." It's human OCR. And it's a massive productivity drain.

How Scan0tron Works

The concept is deceptively simple:

Capture — Select any window or region on your screen
Analyze — AI vision model processes the captured content, understanding context (not just characters)
Auto-fill — Extracted data populates into your target form fields automatically

The key difference from traditional OCR tools: Scan0tron doesn't just read text. It understands context. It knows that "INV-2024-0847" is an invoice number, that "$4,299.00" is a total, and that "Net 30" is payment terms. It maps fields intelligently.

┌─────────────────┐     ┌──────────────┐     ┌─────────────────┐
│  Screen Capture  │ ──→ │  AI Analysis  │ ──→ │  Auto-Fill Form │
│  (any window)    │     │  (contextual) │     │  (your target)  │
└─────────────────┘     └──────────────┘     └─────────────────┘

The Numbers: 50 Tasks Timed

I ran a benchmark: 50 common data entry tasks, comparing manual keyboard entry vs. Scan0tron.

Metric	Manual	Scan0tron
Avg time per task	3.2 min	12 sec
Error rate	4.1%	0.3%
Tasks/hour	18	180+
Time savings	—	94%

The accuracy improvement surprised me most. Humans make typos. AI vision doesn't fat-finger the keyboard.

Why Not Just Use ChatGPT?

ChatGPT can't see your screen. Neither can Gemini, Claude, or any cloud AI. They work with text you give them — they can't observe your desktop, read your open applications, or interact with your forms.

Scan0tron runs locally on your machine. It has direct access to your screen content. No copy-pasting into a chat window. No describing what you're looking at. Just capture → fill.

The Architecture (For the Nerds)

Electron desktop app (cross-platform: Windows, Mac, Linux)
Computer vision pipeline for screen region capture
AI inference for contextual understanding
Form mapping engine that matches extracted fields to target inputs
Zero cloud dependency — works offline, your data never leaves your machine

Pricing That Doesn't Suck

This is where I get opinionated. Most AI tools charge $20-50/month. That's $240-600/year for what is essentially a utility.

Scan0tron is $49. Once. Forever. No subscription. No "free tier" that cripples functionality. No usage caps.

I wrote about why we don't charge monthly fees if you're curious about the business model. TL;DR: Cloudflare Workers + Electron = $0/month hosting costs, so we don't need recurring revenue to stay alive.