DEV Community: Yuvraj Sau

LocalLens — The Unseen AI Ghost Automating Your Browser Behind the Scenes

Yuvraj Sau — Tue, 27 Oct 2026 06:49:10 +0000

*This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine*

What I Built

I built LocalLens, a privacy-first, on-device AI browser extension (MV3) that acts like an invisible digital spirit inside your browser—turning natural language commands into autonomous DOM interactions without transmitting a single byte of screen state to remote servers.

Theme Interpretation & Inspiration

The "Ghost in the Machine" theme represents an invisible, autonomous entity operating silently behind the physical interface. Most AI agents act like cloud phantoms—remote entities hovering over third-party servers, demanding full access to your DOM trees, credentials, and raw screenshots.

LocalLens flips this paradigm. It brings the "ghost" entirely into your own machine. Operating local perception via client-side WebGPU and quantized open-weights, LocalLens silently watches the page, redacts sensitive PII on render like an invisible filter, and executes complex clicks, inputs, and navigations behind the scenes.

I built this for my developer friend Ankit, who needed an intelligent agent to automate repetitive administrative portals and documentation forms—without giving third-party cloud APIs visibility into private keys or student records.

Demo

Code

yuvraj1235 / LocalLens

LocalLens

LocalLens is a privacy-first, locally-deployable browser automation agent. It bridges a Chrome MV3 extension with a Vision-Language Model (VLM) backend to autonomously execute natural language tasks on any webpage (e.g., "Submit the login form").

🏗️ Architecture

The system is split into a robust frontend extension and an intelligent Python backend:

1. Browser Extension (Frontend)

Content Script (content.ts): Runs on the webpage. Walks the DOM, identifies interactive elements, stamps them with stable data-agent-id tags, calculates bounding boxes, and redacts PII (emails, cards, SSNs) client-side before it ever leaves the browser. It also acts as the actuator, executing clicks and typing based on backend commands.
Background Worker (background.js): The secure relay. It captures visible tab screenshots and relays messages between the popup and the content script.
Popup (agentLoop.ts): The orchestration brain. It fetches the UI Graph context from the content script, packages…

View on GitHub

How I Built It

LocalLens is built around a fully local open-source AI architecture that operates entirely within your machine:

In-Browser Perception (WebGPU + ONNX Runtime Web): Lightweight vision and token classification models run directly inside the active browser tab context. The extension maps interactive DOM elements and instantly redacts sensitive PII (emails, authentication headers, API keys) before any action planning takes place.
Local Planning Engine (FastAPI Backend): A lightweight local backend interfaces with open-weight LLMs (such as Qwen 2.5 or Gemma) hosted locally via Ollama or vLLM. It parses natural language intent into structured JSON execution routines.
Execution Engine: Dispatches sanitized JSON action sequences back into the active browser tab context to execute clicks, scrolls, and text inputs automatically.

Why Does Open Innovation Matter?

Open-source AI and client-side runtimes are what allow this "ghost" to stay strictly inside your local machine:

Guaranteed Data Sovereignty: Closed cloud APIs force users to stream private page states over the network. Open-weight models and ONNX WebGPU runtimes keep the entire perception pipeline on-device, guaranteeing zero data exposure.
Predictable $0 Operating Cost: Running open-weight models locally removes API token billing and rate-limiting, allowing continuous, background automation completely free of charge.
Model Flexibility: Users can swap open-weight models based on their hardware specs (e.g., fast 3B parameter models for laptops vs. larger models for workstations).
Offline Capability: Once weights are downloaded, LocalLens can perform structured DOM navigation and form handling completely offline.

My Agent Session

The Spirit of It

The moment LocalLens truly felt like a "ghost in the machine" was during initial testing with Ankit. He typed a prompt into the extension interface, hands completely off the keyboard, and watched as the extension parsed a dense, multi-input administrative form, masked sensitive credentials in real-time, and auto-populated fields step-by-step.

Seeing complex DOM operations unfold locally—with zero network calls to cloud LLMs—felt like watching an invisible assistant control the browser from within the hardware itself.

Prize Categories

Best Use of Gemma (If using Gemma as your local model)
Best Use of GitHub Copilot

LocalLens — Let On-Device AI Do the Browsing Grunt Work for You

Yuvraj Sau — Sat, 24 Oct 2026 14:43:54 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

I built LocalLens, a privacy-first, on-device AI browser extension (MV3) that automates tedious, multi-step web browsing tasks using natural language—so you can sit back, relax, and let your laptop handle the manual clicking.

What it does and what effort it saves

Navigating complex web portals, filling out repetitive multi-page forms, copying structured tables, and sifting through dense administrative panels is draining work. LocalLens turns natural language instructions into precise browser actions (clicks, form inputs, scrolling, table extractions) right inside active browser tabs.

Who gets to be a little lazier because of it?

Anyone tired of manual browser grunt work! I originally built it for my friend Ankit, a developer who spends hours managing internal student portals, staging dashboards, and administrative forms. With LocalLens, Ankit types what he needs done once, hands off the execution to the local agent, and sits back while LocalLens handles the clicks and inputs automatically.

Demo

Code

yuvraj1235 / LocalLens

LocalLens

🏗️ Architecture

The system is split into a robust frontend extension and an intelligent Python backend:

1. Browser Extension (Frontend)

Content Script (content.ts): Runs on the webpage. Walks the DOM, identifies interactive elements, stamps them with stable data-agent-id tags, calculates bounding boxes, and redacts PII (emails, cards, SSNs) client-side before it ever leaves the browser. It also acts as the actuator, executing clicks and typing based on backend commands.
Background Worker (background.js): The secure relay. It captures visible tab screenshots and relays messages between the popup and the content script.
Popup (agentLoop.ts): The orchestration brain. It fetches the UI Graph context from the content script, packages…

View on GitHub

How I Built It

LocalLens is built around a fully local, open-source AI architecture designed to do heavy lifting on-device:

In-Browser Perception (WebGPU + ONNX Runtime Web): Lightweight vision and token classification models run directly inside the client browser tab context. The extension maps interactive DOM components and redacts sensitive PII (emails, authentication headers, phone numbers) before any action planning takes place.
Local Planning Engine (FastAPI Backend): A lightweight local backend interfaces with open-weight LLMs (such as Qwen 2.5 or Gemma) hosted locally via Ollama or vLLM. It parses natural language user intent and turns it into structured JSON execution steps.
Execution Engine: Dispatches sanitized JSON action sequences back into the browser context to perform exact inputs, clicks, and page navigations automatically.

Why Does Open Innovation Matter?

Open-source AI makes "slow and steady" local automation possible without forcing users to sacrifice their data privacy or pay per action:

Absolute Data Sovereignty: Closed cloud APIs require streaming continuous screenshots or raw DOM trees to third-party servers. Open-weight models and ONNX WebGPU runtimes allow the entire perception and planning pipeline to run on-device—meaning personal credentials, student records, and API keys never leave your laptop.
Predictable $0 Operating Cost: Running open-weight models locally eliminates expensive token billing and API usage limits. You can run automated browser tasks all day without spending a dime.
Hardware-Specific Flexibility: Users can swap open-weight models depending on their hardware specs (e.g., fast 3B models for low-power laptops vs. 14B+ parameter models for dedicated workstations).
Air-Gapped & Offline Functionality: Once weights are downloaded, LocalLens can perform structured DOM navigation and form handling completely offline.

The Lazy Payoff

How much time it saves

Manually populating forms, extracting data across tab pages, and navigating internal dashboards usually costs 20–30 minutes of focused effort per task. LocalLens reduces that effort to writing a single prompt and letting the local agent do the walking.

Real-world usage this week

Ankit used LocalLens this week to parse dense documentation panels and auto-fill repetitive local administrative forms. Instead of manually inspecting element states and toggling between tabs, he let LocalLens parse the page layout, sanitize sensitive student IDs on-the-fly, and execute the form-filling pipeline autonomously while he grabbed a cup of coffee.

Prize Categories

Best Use of Gemma (If using Gemma as your local model)
Best Use of GitHub Copilot

Chrome extention

Yuvraj Sau — Wed, 14 Oct 2026 16:48:13 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

LocalLens is a privacy-first, on-device AI browser extension (MV3) that turns natural language instructions into precise web browser actions—without transmitting raw DOM content, screenshots, or sensitive personal data to remote cloud servers.

The Commercial Problem & B2B Utility

Freelancers, compliance managers, and enterprise staff spend hours performing repetitive web interactions—such as populating administrative portals, scraping structured tables, processing invoice dashboards, and executing repetitive web-form workflows. Existing browser AI agents stream unredacted screenshots and DOM trees to external LLM endpoints. For professionals working with medical records, financial documents, API keys, or strict NDA material, using cloud agents presents severe regulatory compliance and privacy risks.

The Side Hustle Angle

LocalLens solves this by offering a zero-leakage, local browser automation tool. Businesses and solo practitioners would willingly pay a monthly subscription or license fee for an enterprise-ready, locally run extension that eliminates privacy liabilities while saving hours of manual browser work every week.

Demo

Code

yuvraj1235 / LocalLens

LocalLens

🏗️ Architecture

The system is split into a robust frontend extension and an intelligent Python backend:

1. Browser Extension (Frontend)

Content Script (content.ts): Runs on the webpage. Walks the DOM, identifies interactive elements, stamps them with stable data-agent-id tags, calculates bounding boxes, and redacts PII (emails, cards, SSNs) client-side before it ever leaves the browser. It also acts as the actuator, executing clicks and typing based on backend commands.
Background Worker (background.js): The secure relay. It captures visible tab screenshots and relays messages between the popup and the content script.
Popup (agentLoop.ts): The orchestration brain. It fetches the UI Graph context from the content script, packages…

View on GitHub

How I Built It

LocalLens is built around a local open-source AI architecture:

In-Browser Perception (WebGPU + ONNX Runtime Web): Lightweight vision and token classification models run directly within the client browser context. The extension analyzes the active tab DOM to map interactive UI nodes and instantly redact sensitive PII (emails, auth headers, numbers) before any action planning takes place.
Local Planning Engine (FastAPI Backend): A lightweight local backend interfaces with open-weight LLMs (such as Qwen 2.5, Gemma, or custom quantized models) hosted locally via Ollama or vLLM. It translates human instructions into structured JSON execution steps.
Execution Engine: Safely dispatches sanitized JSON action sequences back into the browser tab (handling clicks, inputs, scrolls, and navigation).

Why Does Open Innovation Matter?

Open-source AI and client-side web runtimes are the foundation of LocalLens, making capabilities possible that closed APIs simply cannot deliver:

Guaranteed Data Sovereignty: Closed cloud APIs require streaming sensitive page states across the network. Open models and ONNX WebGPU allow the entire perception pipeline to run on-device, guaranteeing zero data exposure.
Predictable $0 Operating Cost: Running open-weight models locally eliminates expensive token billing and rate-limiting, allowing continuous automated web workflows without per-action costs.
Model Flexibility: Users can swap open-weight models based on their hardware specs (e.g., fast 3B models for low-power laptops vs. 14B+ parameter models for dedicated workstations).
Air-Gapped & Offline Functionality: Once weights are fetched, LocalLens can perform structured DOM navigation and form handling completely offline.

My First User

I introduced LocalLens to Ankit, a developer who frequently manages sensitive admin panels, local staging environments, and client records.

Feedback: "The biggest benefit is not needing to sanitize my screen or worry about credentials leaking into third-party prompt logs. It's fast, completely local, and handles repetitive input fields effortlessly."
Willingness to Pay: Ankit noted that freelancers and agencies handling sensitive client data would easily pay $15–$30/month for a client-side agent tool that guarantees zero data leaks and requires no API usage fees.

Prize Categories

Best Use of Gemma (If using Gemma as your local model)
Best Use of GitHub Copilot

LocalLens — On-Device, Privacy-First AI Browser Automation Extension

Yuvraj Sau — Sun, 04 Oct 2026 20:47:33 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

I built LocalLens, a privacy-first, on-device AI browser extension (MV3) that turns natural language instructions into precise web browser actions—without sending raw page content, screenshots, or sensitive data to remote cloud servers.

I built it specifically for my friend Ankit, who regularly interacts with online administrative dashboards, complex web portals, and local developer environments. Ankit wanted an intelligent web assistant to automate repetitive tasks like form auto-filling, table parsing, and navigating dense documentation panels. However, because his daily workflow involves handling sensitive personal credentials, student records, and API keys, uploading screen state or DOM trees to cloud LLMs was a complete non-starter due to privacy risks.

LocalLens solves this by processing visual perception, text extraction, and PII redaction directly inside the browser using client-side WebGPU and local open-source AI.

Demo

How I Built It

LocalLens is engineered around a hybrid local execution pipeline:

Client-Side Perception (WebGPU + ONNX Runtime Web): Lightweight vision and text classification models run directly inside the browser tab context. The extension analyzes web page components locally to identify UI inputs and perform real-time redactions on sensitive PII (emails, auth tokens, phone numbers) before any action planning occurs.
Local Agent Planner (FastAPI Backend): A local Python service accepts sanitized page schemas and routes natural language requests to locally served open-weight models (via Ollama or vLLM running models like Qwen 2.5 or Gemma).
Execution Engine: Converts structured JSON plan outputs back into safe, client-side DOM actions (clicks, key presses, scrolling, form fills) inside the active browser window.

Why Does Open Innovation Matter?

Open-source AI and local runtime standards are the core foundation of LocalLens, making key features possible that closed-source cloud APIs cannot provide:

Complete Privacy & Data Sovereignty: Traditional web agent services require streaming continuous page screenshots or full unredacted DOM representations across the wire. Open-weight models and ONNX WebGPU runtimes allow the entire perception pipeline to live on-device, guaranteeing zero data leakage.
$0 API Costs: Running inference locally on consumer hardware removes token rate limits and recurring usage fees, allowing continuous browser automation for free.
Model Swappability & Custom Fine-Tuning: Closed APIs lock users into monolithic models. With open weights, users can swap models based on their hardware specs (e.g., lightweight models for low-power laptops or larger parameter setups for workstations).
Offline Capability: Core DOM parsing and navigation assistance function entirely without an active internet connection once local weights are downloaded. ## Prize Categories

Build for a Friend
Local First / Open-Source AI