DEV Community

SHYLESH S
SHYLESH S

Posted on

How I Built an Open-Source AI Browser Extension That Actually Executes Tasks


A few months ago I got frustrated with AI tools that could talk about doing things but couldn't actually do them. So I built Cosmos: an open-source Chrome extension where you describe a goal in plain English and watch it get done.

Today I'm open-sourcing it. Here's what it does, how it works, and what I learned building it.


What Cosmos Does

You type a goal — "extract all product prices from this page as JSON" or "fill in this form with my saved details" — and Cosmos:

  1. Reads the live DOM (not a screenshot — the actual structured page)
  2. Sends that + your goal to an LLM and gets back a step-by-step plan
  3. Validates the plan against a strict Zod schema (bad LLM output is rejected, not executed)
  4. Executes each step: click, type, scroll, navigate, extract data
  5. Streams every step back to you live in a side panel

It also remembers. Every run updates a 5-layer memory system so Cosmos gets better at the sites you use most.


The Tech Stack

  • Chrome MV3 — service worker background, content scripts, Side Panel API
  • React 18 + Zustand — the popup/sidepanel UI
  • Dexie (IndexedDB) — the memory layers
  • Zod — plan output validation
  • Vite + Tailwind CSS v3 — build and styling
  • Any OpenAI-compatible API — OpenRouter, Groq, or self-hosted OmniRoute

The Architecture Decision I'm Most Proud Of

Most extension AIs do this:

user types goal → background sends request → content script acts → done
Enter fullscreen mode Exit fullscreen mode

That doesn't work for multi-step tasks. You need to stream progress back, handle cancellation, and let the user approve risky steps mid-flight.

So Cosmos uses a long-lived port (chrome.runtime.connect) between the popup and the background service worker. The orchestrator streams typed events over that port:

type ChatStreamEvent =
  | { kind: "text_chunk"; delta: string }
  | { kind: "status"; stepId: string; status: StepStatus }
  | { kind: "timeline"; steps: PlanStep[] }
  | { kind: "memory_insight"; note: string }
  | { kind: "screenshot_capture"; stepId: string; dataUrl: string }
  | { kind: "token_usage"; estimatedTokens: number }
  | { kind: "done"; success: boolean }
  | { kind: "error"; message: string }
Enter fullscreen mode Exit fullscreen mode

The Zustand store in the popup handles these events and updates the UI. No polling, no request/response ping-pong — just a stream.


The 5-Layer Memory System

The thing that makes Cosmos genuinely useful after a few runs is memory. Each layer serves a different purpose:

Layer What it stores Lifetime
Session Current goal, execution state, active tab Until popup closes
User Preferences, frequent sites, saved form data Persistent
Website Per-domain selector notes, structure hints Persistent
Workflow Named step sequences with usage/success stats Persistent
Failure What went wrong, what recovery worked Persistent

Before every task, Cosmos queries all five layers. The planner prompt includes relevant workflow steps and known failure recovery strategies — so it doesn't repeat mistakes.


The LLM Planner

The planner is the most interesting part to get right. It has three jobs:

  1. Build the prompt — goal + page snapshot (interactive elements, text, forms, tables) + relevant memory + variable context from prior steps
  2. Call the LLM — streamed, so the UI can show typing
  3. Validate the response — Zod schema, before any action runs

The schema enforcement is strict. If the LLM hallucinates an action type that doesn't exist, the whole plan is rejected and the user gets a clear error — not a half-executed mess.

const ActionTypeSchema = z.enum([
  "click", "type", "scroll", "navigate", "select",
  "assert", "extract", "wait",
  "open_tab", "close_tab", "switch_tab",
  "read_clipboard", "write_clipboard",
]);
Enter fullscreen mode Exit fullscreen mode

What I Would Do Differently

1. Start with the port architecture from day one.
I initially tried request/response messaging for everything. Switching to a long-lived port mid-way was painful but completely necessary for streaming.

2. The DOM reader is the most important module.
Everything downstream — the planner, the browser controller — depends on the quality of the page snapshot. I spent more time on the DOM reader than anything else and it was worth it. Giving every interactive element a stable data-cosmos-id attribute before the snapshot is what makes the action executor reliable.

3. Plan validation saves you from yourself.
Early on I skipped validation to move fast. The first time the LLM returned malformed JSON and the extension tried to execute it, I added Zod immediately. It's never been an issue since.


What's Next

The repo is live and open source under MIT:

https://github.com/Cosmos-Agentic-AI/Cosmos_browser_extension

The ROADMAP has a full list of what's planned, including items tagged good first issue if you want to contribute.

The biggest things I want help with:

  • Firefox support (MV2 compatibility)
  • More AI providers (Anthropic Claude, Gemini, Ollama)
  • Real embedding-based memory retrieval

If you build something with it or find a bug, open an issue. I read everything.


Star the repo if this was useful: github.com/Cosmos-Agentic-AI/Cosmos_browser_extension

Top comments (0)