LocalLens — On-Device, Privacy-First AI Browser Automation Extension

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

I built LocalLens, a privacy-first, on-device AI browser extension (MV3) that turns natural language instructions into precise web browser actions—without sending raw page content, screenshots, or sensitive data to remote cloud servers.

I built it specifically for my friend Ankit, who regularly interacts with online administrative dashboards, complex web portals, and local developer environments. Ankit wanted an intelligent web assistant to automate repetitive tasks like form auto-filling, table parsing, and navigating dense documentation panels. However, because his daily workflow involves handling sensitive personal credentials, student records, and API keys, uploading screen state or DOM trees to cloud LLMs was a complete non-starter due to privacy risks.

LocalLens solves this by processing visual perception, text extraction, and PII redaction directly inside the browser using client-side WebGPU and local open-source AI.

Demo

How I Built It

LocalLens is engineered around a hybrid local execution pipeline:

• Client-Side Perception (WebGPU + ONNX Runtime Web): Lightweight vision and text classification models run directly inside the browser tab context. The extension analyzes web page components locally to identify UI inputs and perform real-time redactions on sensitive PII (emails, auth tokens, phone numbers) before any action planning occurs.
• Local Agent Planner (FastAPI Backend): A local Python service accepts sanitized page schemas and routes natural language requests to locally served open-weight models (via Ollama or vLLM running models like Qwen 2.5 or Gemma).
• Execution Engine: Converts structured JSON plan outputs back into safe, client-side DOM actions (clicks, key presses, scrolling, form fills) inside the active browser window.

Why Does Open Innovation Matter?

Open-source AI and local runtime standards are the core foundation of LocalLens, making key features possible that closed-source cloud APIs cannot provide:

1. Complete Privacy & Data Sovereignty: Traditional web agent services require streaming continuous page screenshots or full unredacted DOM representations across the wire. Open-weight models and ONNX WebGPU runtimes allow the entire perception pipeline to live on-device, guaranteeing zero data leakage.
2. $0 API Costs: Running inference locally on consumer hardware removes token rate limits and recurring usage fees, allowing continuous browser automation for free.
3. Model Swappability & Custom Fine-Tuning: Closed APIs lock users into monolithic models. With open weights, users can swap models based on their hardware specs (e.g., lightweight models for low-power laptops or larger parameter setups for workstations).
4. Offline Capability: Core DOM parsing and navigation assistance function entirely without an active internet connection once local weights are downloaded. ## Prize Categories

• Build for a Friend
• Local First / Open-Source AI