A few months ago I got frustrated with AI tools that could talk about doing things but couldn't actually do them. So I built Cosmos: an open-source Chrome extension where you describe a goal in plain English and watch it get done.
Today I'm open-sourcing it. Here's what it does, how it works, and what I learned building it.
What Cosmos Does
You type a goal — "extract all product prices from this page as JSON" or "fill in this form with my saved details" — and Cosmos:
- Reads the live DOM (not a screenshot — the actual structured page)
- Sends that + your goal to an LLM and gets back a step-by-step plan
- Validates the plan against a strict Zod schema (bad LLM output is rejected, not executed)
- Executes each step: click, type, scroll, navigate, extract data
- Streams every step back to you live in a side panel
It also remembers. Every run updates a 5-layer memory system so Cosmos gets better at the sites you use most.
The Tech Stack
- Chrome MV3 — service worker background, content scripts, Side Panel API
- React 18 + Zustand — the popup/sidepanel UI
- Dexie (IndexedDB) — the memory layers
- Zod — plan output validation
- Vite + Tailwind CSS v3 — build and styling
- Any OpenAI-compatible API — OpenRouter, Groq, or self-hosted OmniRoute
The Architecture Decision I'm Most Proud Of
Most extension AIs do this:
user types goal → background sends request → content script acts → done
That doesn't work for multi-step tasks. You need to stream progress back, handle cancellation, and let the user approve risky steps mid-flight.
So Cosmos uses a long-lived port (chrome.runtime.connect) between the popup and the background service worker. The orchestrator streams typed events over that port:
type ChatStreamEvent =
| { kind: "text_chunk"; delta: string }
| { kind: "status"; stepId: string; status: StepStatus }
| { kind: "timeline"; steps: PlanStep[] }
| { kind: "memory_insight"; note: string }
| { kind: "screenshot_capture"; stepId: string; dataUrl: string }
| { kind: "token_usage"; estimatedTokens: number }
| { kind: "done"; success: boolean }
| { kind: "error"; message: string }
The Zustand store in the popup handles these events and updates the UI. No polling, no request/response ping-pong — just a stream.
The 5-Layer Memory System
The thing that makes Cosmos genuinely useful after a few runs is memory. Each layer serves a different purpose:
| Layer | What it stores | Lifetime |
|---|---|---|
| Session | Current goal, execution state, active tab | Until popup closes |
| User | Preferences, frequent sites, saved form data | Persistent |
| Website | Per-domain selector notes, structure hints | Persistent |
| Workflow | Named step sequences with usage/success stats | Persistent |
| Failure | What went wrong, what recovery worked | Persistent |
Before every task, Cosmos queries all five layers. The planner prompt includes relevant workflow steps and known failure recovery strategies — so it doesn't repeat mistakes.
The LLM Planner
The planner is the most interesting part to get right. It has three jobs:
- Build the prompt — goal + page snapshot (interactive elements, text, forms, tables) + relevant memory + variable context from prior steps
- Call the LLM — streamed, so the UI can show typing
- Validate the response — Zod schema, before any action runs
The schema enforcement is strict. If the LLM hallucinates an action type that doesn't exist, the whole plan is rejected and the user gets a clear error — not a half-executed mess.
const ActionTypeSchema = z.enum([
"click", "type", "scroll", "navigate", "select",
"assert", "extract", "wait",
"open_tab", "close_tab", "switch_tab",
"read_clipboard", "write_clipboard",
]);
What I Would Do Differently
1. Start with the port architecture from day one.
I initially tried request/response messaging for everything. Switching to a long-lived port mid-way was painful but completely necessary for streaming.
2. The DOM reader is the most important module.
Everything downstream — the planner, the browser controller — depends on the quality of the page snapshot. I spent more time on the DOM reader than anything else and it was worth it. Giving every interactive element a stable data-cosmos-id attribute before the snapshot is what makes the action executor reliable.
3. Plan validation saves you from yourself.
Early on I skipped validation to move fast. The first time the LLM returned malformed JSON and the extension tried to execute it, I added Zod immediately. It's never been an issue since.
What's Next
The repo is live and open source under MIT:
https://github.com/Cosmos-Agentic-AI/Cosmos_browser_extension
The ROADMAP has a full list of what's planned, including items tagged good first issue if you want to contribute.
The biggest things I want help with:
- Firefox support (MV2 compatibility)
- More AI providers (Anthropic Claude, Gemini, Ollama)
- Real embedding-based memory retrieval
If you build something with it or find a bug, open an issue. I read everything.
Star the repo if this was useful: github.com/Cosmos-Agentic-AI/Cosmos_browser_extension

Top comments (0)