This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady
What I Built
I built LocalLens, a privacy-first, on-device AI browser extension (MV3) that automates tedious, multi-step web browsing tasks using natural language—so you can sit back, relax, and let your laptop handle the manual clicking.
What it does and what effort it saves
Navigating complex web portals, filling out repetitive multi-page forms, copying structured tables, and sifting through dense administrative panels is draining work. LocalLens turns natural language instructions into precise browser actions (clicks, form inputs, scrolling, table extractions) right inside active browser tabs.
Who gets to be a little lazier because of it?
Anyone tired of manual browser grunt work! I originally built it for my friend Ankit, a developer who spends hours managing internal student portals, staging dashboards, and administrative forms. With LocalLens, Ankit types what he needs done once, hands off the execution to the local agent, and sits back while LocalLens handles the clicks and inputs automatically.
Demo
Code
LocalLens
LocalLens is a privacy-first, locally-deployable browser automation agent. It bridges a Chrome MV3 extension with a Vision-Language Model (VLM) backend to autonomously execute natural language tasks on any webpage (e.g., "Submit the login form").
🏗️ Architecture
The system is split into a robust frontend extension and an intelligent Python backend:
1. Browser Extension (Frontend)
-
Content Script (
content.ts): Runs on the webpage. Walks the DOM, identifies interactive elements, stamps them with stabledata-agent-idtags, calculates bounding boxes, and redacts PII (emails, cards, SSNs) client-side before it ever leaves the browser. It also acts as the actuator, executing clicks and typing based on backend commands. -
Background Worker (
background.js): The secure relay. It captures visible tab screenshots and relays messages between the popup and the content script. -
Popup (
agentLoop.ts): The orchestration brain. It fetches the UI Graph context from the content script, packages…
How I Built It
LocalLens is built around a fully local, open-source AI architecture designed to do heavy lifting on-device:
- In-Browser Perception (WebGPU + ONNX Runtime Web): Lightweight vision and token classification models run directly inside the client browser tab context. The extension maps interactive DOM components and redacts sensitive PII (emails, authentication headers, phone numbers) before any action planning takes place.
- Local Planning Engine (FastAPI Backend): A lightweight local backend interfaces with open-weight LLMs (such as Qwen 2.5 or Gemma) hosted locally via Ollama or vLLM. It parses natural language user intent and turns it into structured JSON execution steps.
- Execution Engine: Dispatches sanitized JSON action sequences back into the browser context to perform exact inputs, clicks, and page navigations automatically.
Why Does Open Innovation Matter?
Open-source AI makes "slow and steady" local automation possible without forcing users to sacrifice their data privacy or pay per action:
- Absolute Data Sovereignty: Closed cloud APIs require streaming continuous screenshots or raw DOM trees to third-party servers. Open-weight models and ONNX WebGPU runtimes allow the entire perception and planning pipeline to run on-device—meaning personal credentials, student records, and API keys never leave your laptop.
- Predictable $0 Operating Cost: Running open-weight models locally eliminates expensive token billing and API usage limits. You can run automated browser tasks all day without spending a dime.
- Hardware-Specific Flexibility: Users can swap open-weight models depending on their hardware specs (e.g., fast 3B models for low-power laptops vs. 14B+ parameter models for dedicated workstations).
- Air-Gapped & Offline Functionality: Once weights are downloaded, LocalLens can perform structured DOM navigation and form handling completely offline.
The Lazy Payoff
How much time it saves
Manually populating forms, extracting data across tab pages, and navigating internal dashboards usually costs 20–30 minutes of focused effort per task. LocalLens reduces that effort to writing a single prompt and letting the local agent do the walking.
Real-world usage this week
Ankit used LocalLens this week to parse dense documentation panels and auto-fill repetitive local administrative forms. Instead of manually inspecting element states and toggling between tabs, he let LocalLens parse the page layout, sanitize sensitive student IDs on-the-fly, and execute the form-filling pipeline autonomously while he grabbed a cup of coffee.
Prize Categories
- Best Use of Gemma (If using Gemma as your local model)
- Best Use of GitHub Copilot
Top comments (0)