What I Built

LocalLens is a privacy-first, on-device AI browser extension (MV3) that turns natural language instructions into precise web browser actions—without transmitting raw DOM content, screenshots, or sensitive personal data to remote cloud servers.

The Commercial Problem & B2B Utility

Freelancers, compliance managers, and enterprise staff spend hours performing repetitive web interactions—such as populating administrative portals, scraping structured tables, processing invoice dashboards, and executing repetitive web-form workflows. Existing browser AI agents stream unredacted screenshots and DOM trees to external LLM endpoints. For professionals working with medical records, financial documents, API keys, or strict NDA material, using cloud agents presents severe regulatory compliance and privacy risks.

The Side Hustle Angle

LocalLens solves this by offering a zero-leakage, local browser automation tool. Businesses and solo practitioners would willingly pay a monthly subscription or license fee for an enterprise-ready, locally run extension that eliminates privacy liabilities while saving hours of manual browser work every week.

Code

yuvraj1235 / LocalLens

LocalLens

LocalLens is a privacy-first, locally-deployable browser automation agent. It bridges a Chrome MV3 extension with a Vision-Language Model (VLM) backend to autonomously execute natural language tasks on any webpage (e.g., "Submit the login form").

🏗️ Architecture

The system is split into a robust frontend extension and an intelligent Python backend:

1. Browser Extension (Frontend)

Content Script (content.ts): Runs on the webpage. Walks the DOM, identifies interactive elements, stamps them with stable data-agent-id tags, calculates bounding boxes, and redacts PII (emails, cards, SSNs) client-side before it ever leaves the browser. It also acts as the actuator, executing clicks and typing based on backend commands.
Background Worker (background.js): The secure relay. It captures visible tab screenshots and relays messages between the popup and the content script.
Popup (agentLoop.ts): The orchestration brain. It fetches the UI Graph context from the content script, packages…

View on GitHub

How I Built It

LocalLens is built around a local open-source AI architecture:

In-Browser Perception (WebGPU + ONNX Runtime Web): Lightweight vision and token classification models run directly within the client browser context. The extension analyzes the active tab DOM to map interactive UI nodes and instantly redact sensitive PII (emails, auth headers, numbers) before any action planning takes place.

Local Planning Engine (FastAPI Backend): A lightweight local backend interfaces with open-weight LLMs (such as Qwen 2.5, Gemma, or custom quantized models) hosted locally via Ollama or vLLM. It translates human instructions into structured JSON execution steps.

Execution Engine: Safely dispatches sanitized JSON action sequences back into the browser tab (handling clicks, inputs, scrolls, and navigation).

Why Does Open Innovation Matter?

Open-source AI and client-side web runtimes are the foundation of LocalLens, making capabilities possible that closed APIs simply cannot deliver:

Guaranteed Data Sovereignty: Closed cloud APIs require streaming sensitive page states across the network. Open models and ONNX WebGPU allow the entire perception pipeline to run on-device, guaranteeing zero data exposure.

Predictable $0 Operating Cost: Running open-weight models locally eliminates expensive token billing and rate-limiting, allowing continuous automated web workflows without per-action costs.

Model Flexibility: Users can swap open-weight models based on their hardware specs (e.g., fast 3B models for low-power laptops vs. 14B+ parameter models for dedicated workstations).

Air-Gapped & Offline Functionality: Once weights are fetched, LocalLens can perform structured DOM navigation and form handling completely offline.

My First User

I introduced LocalLens to Ankit, a developer who frequently manages sensitive admin panels, local staging environments, and client records.

Feedback: "The biggest benefit is not needing to sanitize my screen or worry about credentials leaking into third-party prompt logs. It's fast, completely local, and handles repetitive input fields effortlessly."

Willingness to Pay: Ankit noted that freelancers and agencies handling sensitive client data would easily pay $15–$30/month for a client-side agent tool that guarantees zero data leaks and requires no API usage fees.

DEV Community

Chrome extention