DEV Community

Cover image for From Scraping to the "RSS for AI": Anatomy of an Agentic Front-End
Quentin Merle
Quentin Merle

Posted on Originally published at vibrisse-studio.dev on

From Scraping to the "RSS for AI": Anatomy of an Agentic Front-End

#ai #webdev #frontend #architecture

🧭 Quick Links & Lab Resources:

To let an AI agent buy shoes on a website today, the tech industry came up with a strange solution: start a 1.2 GB headless Chromium browser in the background, take screenshots every two seconds, and hope a vision model clicks the right button.

It is slow (two seconds per click), wastes massive server resources, and breaks completely the moment a developer renames a CSS class in production.

At the same time, for human users, companies are trying to squeeze all of e-commerce into a chat box. We are expected to buy a $180 pair of running shoes by typing prompts, completely blind, as if screens were stuck in the 1970s.

The agentic web is caught in an awkward trap: slow visual scraping on one side, and blind chat prompts on the other.

The Front-End is not going away: it is becoming the native place where AI actually runs.

Here is an honest breakdown of our production test lab (Kuro Runner V1), what broke in the real world, and the 3-tier architecture we built to fix it.

1. The Core Problem: Two Dead-Ends

The Waste of Headless Scraping

Why do AI agents struggle to buy products on regular websites? Because we force them to read the screen visually instead of giving them access to the data already sitting in browser memory.

Running a headless browser just to click "Add to Cart" is like using a sledgehammer to crack a nut:

  • 1,260 to 1,325 MB of RAM per session.
  • 1,100 to 1,850 ms of cold start time just to boot the browser engine.
  • Breaks easily: any CSS update breaks the visual selectors. We spent fifteen years optimizing web performance and loading times. Spawning hidden browsers on our servers just to click buttons makes no sense.

Why Pure Text Fails

On the other side, trying to do everything inside a chat bubble is just as bad.

Natural language is great for general questions:
“Find me lightweight trail shoes for wide feet with good cushioning.”

But text is terrible when you need to see colors, materials, or how shoes actually look.

Picking between Obsidian Black and Raw Chalk by reading text descriptions in a prompt box is a huge step backward. Nobody buys sneakers without looking at them.

Visual UI should not disappear: it should just appear when needed (Just-in-Time UI), during the exact moment a human makes a decision.

2. The Architectural Solution: 3-Tier Progressive Enhancement

To build a clean system, we followed a classic web principle: Progressive Enhancement.

Illustration

We need to support two different setups:

Illustration

Tier 1: The "RSS for AI" (**/agent-feed.json**)

In the early 2000s, RSS feeds allowed readers to get blog posts without scraping HTML. Today, we can do the exact same thing for AI agents: expose a static JSON feed (/agent-feed.json) linked in the page header.

{
  "$schema": "https://json-schemas.org/draft/2020-12/schema",
  "version": "1.0.0",
  "name": "Kuro Footwear Store",
  "tools": [
    {
      "name": "order_kuro_shoe",
      "description": "Configures Kuro Runner V1 shoes and prepares checkout",
      "parameters": {
        "type": "object",
        "properties": {
          "size": { "type": "number", "enum": [7, 8, 9, 10, 11] },
          "color": { "type": "string", "enum": ["Obsidian Black", "Deep Indigo", "Raw Chalk"] }
        },
        "required": ["size", "color"]
      },
      "handoff": {
        "type": "checkout-deep-link",
        "template": "https://kuro-agentic-web.vercel.app/checkout?sku=KURO-V1-{color}-{size}&source=agent_feed"
      }
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode
  • The numbers: One quick HTTP GET request of 1.75 KB (751 bytes compressed, 50 ms on an Edge CDN, zero browser RAM).
  • Why it works so well: It is 100% universal today. Any LLM that can make HTTP calls (ChatGPT, Claude, Cursor, Antigravity) reads this file and asks the right questions: “Which size and color would you like?” ### Tier 2: The Sandboxed Micro-UI (MCP Apps)

Text is fine, but users want to see the product. We declare an interface entry point:

"ui": {
  "type": "mcp-app",
  "entrypoint": "https://kuro-agentic-web.vercel.app/widget.html",
  "tag": "kuro-configurator",
  "format": "text/html;profile=mcp-app"
}
Enter fullscreen mode Exit fullscreen mode

Security in Chat Clients

If you try to paste this into ChatGPT or Claude.ai, no widget shows up. You just get text.

Why? Because of a basic security rule: Content Security Policies (**frame-src 'self'**).
If chat apps allowed an AI model to load external iframes directly, hackers could easily inject malicious links, show fake login forms, or steal user cookies.

That is why the MCP Apps (SEP-1865) specification was created:

  1. The chat app downloads the HTML/JS code.
  2. It runs it inside an isolated sandbox (sandbox="allow-scripts" with no network access).
  3. The widget talks to the chat strictly through typed messages (postMessage via JSON-RPC 2.0). ### What Broke: When 3D Crashes the Chat

At first, I tried putting our full Three.js 3D configurator directly inside the chat widget (3D model, mouse rotation, studio lighting, 8 MB file).

On my Mac, it ran at 60 fps without any issues.

But inside real sandboxed webviews used by AI agents, it hit three walls:

  1. No GPU acceleration: In many agent webviews, WebGL2 is disabled or limited. The canvas stayed completely black.
  2. Blocked network calls: To prevent data leaks, the sandbox blocks network requests (connect-src 'none'). Downloading the 8 MB 3D file failed.
  3. Out of memory: In a long conversation, opening multiple 3D models quickly uses up all GPU memory, causing the tab to crash. #### The Simple Fix: Snapshot Baking

When 3D fails, you find a smarter, lighter way.

During the build process, we pre-rendered clean studio photos of the shoes from the best angle for each color. We compressed them as high-quality JPEGs and included them directly in a small 112 KB component.

[The Three Tiers]
Tier 1: Plain text (Universal · 0 KB assets · Works on 100% of LLMs)
   │
Tier 2: Snapshot Baking (Production Standard · 112 KB · No WebGL lag · 0 MB GPU memory)
   │
Tier 3: Full Three.js 3D (Optional bonus when WebGL2 is available)
Enter fullscreen mode Exit fullscreen mode
  • The result:
    • 0 ms loading delay.
    • 0 MB of GPU memory used.
    • Zero blocked network calls.
    • To the user, it looks just as good as real-time 3D, but it never crashes. ### 4 Golden Rules for Chat UIs

Building a component for a chat stream is very different from building a normal web page:

1. Keep it Small (The Micro-Viewport)

Chat sidebars are narrow (380 to 500 px wide).
Remove all extra clutter: no big brand logos, no hamburger menus, no cookie banners. Show only the product, the options (size, color), and the action button.

2. The Mouse Wheel Trap

This is an annoying bug: the user tries to scroll up to re-read a message, but their mouse cursor moves over your widget.
If your component captures scroll events, the chat stops scrolling, and the user gets stuck zooming in on the shoe laces.
Rule: Set a max height (max-height: 480px), disable scroll hijacking (touch-action: pan-y), and require an explicit click before capturing the mouse wheel.

3. Match the Host Theme (No Blinding White Screens)

Nothing looks worse than a bright white iframe popping up inside an IDE or chat app set to Dark Mode. Your component should use host CSS variables (--host-bg-primary, --host-text-primary) so it fits in naturally.

4. The Receipt State (Stopping Zombie Buttons)

On regular websites, after you pay, you leave the page.
In a chat app, the widget stays in the chat history forever.
What happens if the user scrolls up three days later and accidentally clicks "Order" again?
The Receipt Pattern: As soon as the user confirms, the widget disables its buttons and turns into a simple, permanent receipt (“Kuro Runner V1 — Deep Indigo, US 9 · Order confirmed ✅”).

One Component for Web and Agents (**<kuro-configurator>**)

To avoid writing and maintaining two separate front-end codebases, we use a single Web Component:

// Web Component for both normal web and AI chat
class KuroConfigurator extends HTMLElement {
  connectedCallback() {
    this.context = this.getAttribute("context") || "web";
    this.attachShadow({ mode: "open" });
    this.render();
  }

  async handleOrder(selection) {
    if (this.context === "agent") {
      // In chat: send a message to the chat host
      window.parent.postMessage({
        jsonrpc: "2.0",
        method: "ui/notifications/action",
        params: { action: "order_kuro_shoe", data: selection }
      }, "*");
    } else {
      // On regular web: update the shopping cart directly
      await window.__SHOP_STORE__.addToCart(selection);
    }
  }
}
customElements.define("kuro-configurator", KuroConfigurator);
Enter fullscreen mode Exit fullscreen mode

One component, one place for business logic, two ways to use it.

3. The Takeaway: What This Unlocks

Benchmark Numbers

(Tested on Apple M1 Pro, 32 GB RAM, Chromium Headless, and Vercel Edge CDN)

Metric 1. Headless Scraping 2. In-Browser WebMCP 3. RSS Feed + Micro-UI
Discovery Time 1,100 to 1,850 ms (Booting Chromium) < 1 ms (Direct memory access) 45 to 65 ms (Edge CDN)
RAM Usage 1,260 to 1,325 MB 0 MB extra 0 MB browser RAM
Ingested Data 40.6 KB HTML (~10,150 tokens) 0 tokens 1.75 KB (~440 tokens)
Build Stability Breaks whenever CSS changes 100% stable (Typed schema) 100% stable (Versioned schema)
Visual Result Static PNG screenshot Full active tab Fast Micro-UI (< 10 ms, 112 KB)
Payment Safety Risky (Credit card shared with AI) Active user session 1-click biometric confirmation

Two Real-World Questions

1. Why would online stores allow this?

Why would an e-commerce brand let users buy inside a chat instead of visiting their store?
For the exact same reason restaurants joined Google Maps and food delivery apps: stores follow where the buyers are. If competitors make buying easier, everyone has to adapt.

2. The Security Risk (The Confused Deputy)

Giving an AI agent permission to run code on a tab where you are logged in is risky.
If the agent reads a malicious website with hidden prompt injection, it could try to make orders using your account without your permission.
That is why human confirmation must always be required.

Paying Safely: "The Agent Prepares, The Human Signs"

Giving your credit card numbers directly to an AI model to make payments in the background is a terrible idea and violates basic security rules.

The right way to do it uses biometric confirmation (Passkeys, Apple Pay, Touch ID / Face ID):

  1. The agent configures the order details.
  2. The widget locks into a clean Receipt State.
  3. The user taps Touch ID or uses Face ID to pay. No credit card numbers are ever shared with the AI. No broken sessions. No 12-field checkout forms.

“The agent prepares the cart; the human validates the signature.”

Conclusion: The New Role of Front-End Developers

Some people say AI agents will kill front-end development and replace everything with text prompts.

Our tests prove the exact opposite. What AI replaces are the worst parts of the web: messy checkout forms, cluttered filter sidebars, and confusing menus.

Whenever users need to see colors, inspect products, or safely confirm a payment, clear graphical UI remains essential.

Our job as front-end developers is not disappearing—it is evolving. We are no longer building closed-off website silos. We are building clean, distributable UI components that machines can understand and humans can easily use.

Proudly crafted in Beauce, Québec 🇨🇦. Interested in the alliance between immersive web engineering and local AI sovereignty? Let's connect on LinkedIn or via Vibrisse Studio!

Top comments (0)