<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: SHYLESH S</title>
    <description>The latest articles on DEV Community by SHYLESH S (@shylesh1640).</description>
    <link>https://dev.to/shylesh1640</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3419433%2Ffe40d9eb-739b-43e2-aca8-e49f7ced954a.png</url>
      <title>DEV Community: SHYLESH S</title>
      <link>https://dev.to/shylesh1640</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/shylesh1640"/>
    <language>en</language>
    <item>
      <title>How I Built an Open-Source AI Browser Extension That Actually Executes Tasks</title>
      <dc:creator>SHYLESH S</dc:creator>
      <pubDate>Tue, 17 Nov 2026 12:26:56 +0000</pubDate>
      <link>https://dev.to/shylesh1640/how-i-built-an-open-source-ai-browser-extension-that-actually-executes-tasks-l42</link>
      <guid>https://dev.to/shylesh1640/how-i-built-an-open-source-ai-browser-extension-that-actually-executes-tasks-l42</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fil4gtn9rvqaxhvakiaxp.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fil4gtn9rvqaxhvakiaxp.gif" alt=" " width="700" height="394"&gt;&lt;/a&gt;&lt;/p&gt;




&lt;p&gt;A few months ago I got frustrated with AI tools that could &lt;em&gt;talk&lt;/em&gt; about doing things but couldn't actually &lt;em&gt;do&lt;/em&gt; them. So I built Cosmos: an open-source Chrome extension where you describe a goal in plain English and watch it get done.&lt;/p&gt;

&lt;p&gt;Today I'm open-sourcing it. Here's what it does, how it works, and what I learned building it.&lt;/p&gt;




&lt;h2&gt;
  
  
  What Cosmos Does
&lt;/h2&gt;

&lt;p&gt;You type a goal — "extract all product prices from this page as JSON" or "fill in this form with my saved details" — and Cosmos:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Reads the live DOM (not a screenshot — the actual structured page)&lt;/li&gt;
&lt;li&gt;Sends that + your goal to an LLM and gets back a step-by-step plan&lt;/li&gt;
&lt;li&gt;Validates the plan against a strict Zod schema (bad LLM output is rejected, not executed)&lt;/li&gt;
&lt;li&gt;Executes each step: click, type, scroll, navigate, extract data&lt;/li&gt;
&lt;li&gt;Streams every step back to you live in a side panel&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;It also remembers. Every run updates a 5-layer memory system so Cosmos gets better at the sites you use most.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Tech Stack
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Chrome MV3&lt;/strong&gt; — service worker background, content scripts, Side Panel API&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;React 18 + Zustand&lt;/strong&gt; — the popup/sidepanel UI&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dexie (IndexedDB)&lt;/strong&gt; — the memory layers&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Zod&lt;/strong&gt; — plan output validation&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Vite + Tailwind CSS v3&lt;/strong&gt; — build and styling&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Any OpenAI-compatible API&lt;/strong&gt; — OpenRouter, Groq, or self-hosted OmniRoute&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  The Architecture Decision I'm Most Proud Of
&lt;/h2&gt;

&lt;p&gt;Most extension AIs do this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;user types goal → background sends request → content script acts → done
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That doesn't work for multi-step tasks. You need to stream progress back, handle cancellation, and let the user approve risky steps mid-flight.&lt;/p&gt;

&lt;p&gt;So Cosmos uses a &lt;strong&gt;long-lived port&lt;/strong&gt; (&lt;code&gt;chrome.runtime.connect&lt;/code&gt;) between the popup and the background service worker. The orchestrator streams typed events over that port:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="nc"&gt;ChatStreamEvent&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
  &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"text_chunk"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;delta&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;stepId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;StepStatus&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"timeline"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;steps&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;PlanStep&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"memory_insight"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;note&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"screenshot_capture"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;stepId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;dataUrl&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"token_usage"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;estimatedTokens&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;number&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"done"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;success&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;boolean&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"error"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The Zustand store in the popup handles these events and updates the UI. No polling, no request/response ping-pong — just a stream.&lt;/p&gt;




&lt;h2&gt;
  
  
  The 5-Layer Memory System
&lt;/h2&gt;

&lt;p&gt;The thing that makes Cosmos genuinely useful after a few runs is memory. Each layer serves a different purpose:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;What it stores&lt;/th&gt;
&lt;th&gt;Lifetime&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Session&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Current goal, execution state, active tab&lt;/td&gt;
&lt;td&gt;Until popup closes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;User&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Preferences, frequent sites, saved form data&lt;/td&gt;
&lt;td&gt;Persistent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Website&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Per-domain selector notes, structure hints&lt;/td&gt;
&lt;td&gt;Persistent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Workflow&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Named step sequences with usage/success stats&lt;/td&gt;
&lt;td&gt;Persistent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Failure&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;What went wrong, what recovery worked&lt;/td&gt;
&lt;td&gt;Persistent&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Before every task, Cosmos queries all five layers. The planner prompt includes relevant workflow steps and known failure recovery strategies — so it doesn't repeat mistakes.&lt;/p&gt;




&lt;h2&gt;
  
  
  The LLM Planner
&lt;/h2&gt;

&lt;p&gt;The planner is the most interesting part to get right. It has three jobs:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Build the prompt&lt;/strong&gt; — goal + page snapshot (interactive elements, text, forms, tables) + relevant memory + variable context from prior steps&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Call the LLM&lt;/strong&gt; — streamed, so the UI can show typing&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Validate the response&lt;/strong&gt; — Zod schema, before any action runs&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The schema enforcement is strict. If the LLM hallucinates an action type that doesn't exist, the whole plan is rejected and the user gets a clear error — not a half-executed mess.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nc"&gt;ActionTypeSchema&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;z&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kd"&gt;enum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
  &lt;span class="s2"&gt;"click"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"scroll"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"navigate"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"select"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;"assert"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"extract"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"wait"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;"open_tab"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"close_tab"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"switch_tab"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="s2"&gt;"read_clipboard"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"write_clipboard"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  What I Would Do Differently
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Start with the port architecture from day one.&lt;/strong&gt;&lt;br&gt;
I initially tried request/response messaging for everything. Switching to a long-lived port mid-way was painful but completely necessary for streaming.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. The DOM reader is the most important module.&lt;/strong&gt;&lt;br&gt;
Everything downstream — the planner, the browser controller — depends on the quality of the page snapshot. I spent more time on the DOM reader than anything else and it was worth it. Giving every interactive element a stable &lt;code&gt;data-cosmos-id&lt;/code&gt; attribute before the snapshot is what makes the action executor reliable.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Plan validation saves you from yourself.&lt;/strong&gt;&lt;br&gt;
Early on I skipped validation to move fast. The first time the LLM returned malformed JSON and the extension tried to execute it, I added Zod immediately. It's never been an issue since.&lt;/p&gt;




&lt;h2&gt;
  
  
  What's Next
&lt;/h2&gt;

&lt;p&gt;The repo is live and open source under MIT:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://github.com/Cosmos-Agentic-AI/Cosmos_browser_extension" rel="noopener noreferrer"&gt;https://github.com/Cosmos-Agentic-AI/Cosmos_browser_extension&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://github.com/Cosmos-Agentic-AI/Cosmos_browser_extension/blob/main/ROADMAP.md" rel="noopener noreferrer"&gt;ROADMAP&lt;/a&gt; has a full list of what's planned, including items tagged &lt;code&gt;good first issue&lt;/code&gt; if you want to contribute.&lt;/p&gt;

&lt;p&gt;The biggest things I want help with:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Firefox support&lt;/strong&gt; (MV2 compatibility)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;More AI providers&lt;/strong&gt; (Anthropic Claude, Gemini, Ollama)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Real embedding-based memory retrieval&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you build something with it or find a bug, open an issue. I read everything.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Star the repo if this was useful: &lt;a href="https://github.com/Cosmos-Agentic-AI/Cosmos_browser_extension" rel="noopener noreferrer"&gt;github.com/Cosmos-Agentic-AI/Cosmos_browser_extension&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

</description>
      <category>extensions</category>
      <category>ai</category>
      <category>opensource</category>
      <category>javascript</category>
    </item>
  </channel>
</rss>
