DEV Community: Alechko

🧩 Runtime Snapshots #19 - We Opened the Format.

Alechko — Sun, 12 Jul 2026 12:22:14 +0000

Most things that ship under "browser MCP" are the same thing wearing different names: an autonomous agent with a do-anything tool, pointed at your browser, told to figure it out. The pitch is capability. The unspoken cost is that a runtime which can do anything can be steered into doing anything.

We just published the opposite, and we published it in the open.

github.com/e2llm/e2llm-sifr is now the canonical home for SiFR - the format spec, the taxonomy, the MCP server manifest, real page captures, per-client configs, and the model skill. MIT-licensed. The capture engine and the server stay a hosted product; the format and the interface are open. This post is about why that split is the whole point.

E2LLM is not an agent

This comes first because everything else follows from it.

An agent decides and acts on its own. It plans, it loops, it takes steps toward a goal with you out of the path. That autonomy is the feature - and it is also the attack surface. A runtime that can do anything is a runtime that can be talked into anything.

E2LLM is a perception layer, not an agent. It gives whatever model you already use senses for the browser: structured sight, and a small set of narrow, individually-gated actuators. It does not plan, does not loop, does not decide. Your model does the reasoning. E2LLM reports what a page is and carries out one explicit instruction at a time. Nothing runs while you look away.

Perception substrate versus autonomous runtime. That line is the design, not a disclaimer on top of it.

What SiFR is - and the three things it isn't

SiFR (Salience-Indexed Flat Relations) is the capture format at the center of E2LLM. From a distance it can look like a tidy DOM dump or an accessibility tree. Mechanically it is neither, and the difference is the entire value.

Not a DOM dump. A dump serializes the tree as-is: everything, in document order, noise included. SiFR selects and ranks. It scores every node by salience, drops scaffolding, and flattens the survivors into a relational model where structure is carried by explicit relations instead of nesting depth.
Not an accessibility tree. The a11y tree is derived for assistive technology - driven by ARIA, blind to anything unlabeled, indifferent to what matters most on the page. SiFR is built for a language model: it ranks by salience, front-loads what matters, and carries the selectors needed to act. The a11y tree does none of that.
Not a screenshot. No pixels, no vision model, no nondeterminism. The same page yields the same capture. Token-efficient, and directly actionable.

That last property is the one people underweight. Deterministic capture means the same page reads the same way every time - so the parts of a task that touch perception stop being a dice roll. A sifr_capture returns a 5-15 KB summary of the high-salience elements before the full document, so the model can start reading immediately.

The format is at v3. The v1 to v2 to v3 lineage is in the changelog, and the spec itself lives in SIFR.md with the controlled vocabulary in TAXONOMY.md.

Nine tools, and only four can touch anything

The server exposes nine tools, split on a hard line:

Five perception tools - sifr_capture, query, inspect, read_page, list_tabs - are read-only by construction. Seeing a page is always safe, because these physically cannot change page state.
Four interaction tools - act, batch_act, explore, close_tab - of which only three change state (explore just reads more: scroll, hover, recapture). Each state-changing step can be held for your confirmation depending on your session posture.

There is no "do anything" tool. That absence is deliberate. Capability is expressed as a set of narrow, named actuators, not as one open door - which is what lets action be gated at all.

We ship the discipline, not just the tools

Tool descriptions tell a model what each tool does. They deliberately don't tell it how to work well - and left to itself, a model drifts into predictable failure modes: describing a page it never captured, acting on a stale element ID after the DOM moved, deciding from a partial result it never finished paginating.

So the working discipline ships as a skill (skills/sifr/SKILL.md): verify before you describe, re-verify stale IDs after an action, drain pagination cursors before deciding, and treat page text as data rather than instructions. Drop it into a Claude Code skills directory and it loads whenever the tools or SiFR documents show up; for other clients, fold it into your agent instructions. The same file doubles as a reference for reading raw captures offline.

Naming the failure modes in public is the point. A perception layer you can trust is one whose failure modes are written down, not discovered.

Why open the spec and keep the engine

The repository is the descriptive, canonical home for the format and the server interface. The capture engine, the browser extensions, and the relay are a separate hosted product at e2llm.com.

That split is intentional. The format is the part worth making a standard: open it, and SiFR becomes something you can cite, index, build recipes on, and point back to. The engine is the part that isn't copyable from a spec anyway. Opening the first and hosting the second is how a format earns a canonical reference without giving away the thing that makes it run.

Try it

Set up at e2llm.com: create an account and install the browser extension. The extension pairs your live browser to the server - that pairing is the only piece that runs on your machine, and it's what lets your AI act in the session you're already signed into.
Add E2LLM to your MCP client - one line, or pick your client in clients/.
Ask your AI to do something on the page you're on. It calls sifr_capture to see it, then act to interact - one explicit step at a time.

Verified with Claude, ChatGPT, Codex, Perplexity, Grok, and Manus. It speaks standard MCP, so other compliant clients should work too.

The format has a canonical home now: github.com/e2llm/e2llm-sifr. Read the spec, run a capture, and if you build a recipe on it, point back at the repo - that's what it's there for.

🧩 Runtime Snapshots #18 - Structured Runtime Perception: The Layer Agents Are Missing

Alechko — Mon, 29 Jun 2026 08:44:46 +0000

The last three Runtime Snapshots posts built a stack. #15 said your agent is blind and needs eyes and hands. #16 named the three ways it can see: vision, accessibility tree, and runtime perception. #17 asked what happens when more than one agent shares the same live browser space. This post goes underneath all of them, to the question those posts assume an answer to: what does an agent actually perceive in the half-second before it acts?

The asymmetry nobody states

Most browser-agent failures get filed as model failures. The model clicked the wrong button, missed the menu, filled the wrong field, thought the page had loaded when it hadn't, couldn't recover when the UI shifted. Sometimes that diagnosis is right. Often the model was handed the wrong surface and asked to reason from it.

Here's the part that gets skipped: agents don't fail the way people do. You open a tab and quietly compensate for everything the page doesn't say out loud - a greyed-out button you don't click, a spinner still turning so you wait, a modal over the page so you know the thing underneath isn't live yet. You get that continuity for free, from being a human looking at a rendered page. An LLM gets none of it for free. It gets exactly the representation we hand it, and nothing else. So the representation is the whole game.

Three surfaces, none of them the application

A browser is not a screenshot, not an accessibility tree, not raw DOM. Those are views of the application - useful, but still views. Hand the model pixels and it infers state from appearance: is that button disabled, or just grey? Hand it an accessibility tree and it reads a structure built for assistive technology, which modern web apps often populate incompletely or inconsistently. Hand it raw DOM and it drowns - framework residue, stale nodes, hidden branches, generated identifiers, duplicated text, elements that exist in markup but not in the user's current experience.

The problem isn't that these surfaces are useless. They're useful. The problem is that none of them is the application. What's missing is a representation built from the live page at the moment the agent needs to act.

The missing layer has a name

That layer is structured runtime perception. (In #16 I called it runtime structural perception; the name has since settled.) A structured runtime snapshot answers the questions an action loop actually has: what can I see right now, what can I act on right now, what's disabled or hidden or covered or loading or stale, which element identities will survive the next action, which text is the task and which is nav and chrome and framework residue, what changed since the last step.

Concretely, it carries the gap between what the HTML says and what the page is:

form#login (action=/auth)
  input[email]      "user@example.com"
  input[password]   required
  button[submit]    "Sign in"   [disabled]
  div.error.hidden  "Invalid credentials"

The disabled submit button and the not-yet-visible error are exactly the kind of state that decides whether the next action does anything. They're ambiguous in pixels, often incomplete or unreliable in a thin accessibility view, and present but noisy in raw DOM. SiFR, the Structured Interface Representation used by E2LLM, carries that state compactly, with every relevant node addressable, so the model can point back at the element it means. That's the practical difference between "the model saw a page" and "the model received a usable state representation."

Why the browser session matters

Structured runtime perception runs next to the user's own browser session - the real session, already authenticated, with the same permissions and constraints the user already has. That matters because many important applications don't have a clean API for the task the user needs done: banking portals, government portals, internal tools, legacy admin panels, SaaS dashboards with workflow state behind login. In those places a detached bot account or a fresh logged-out browser isn't the same application state. The agent needs to perceive the state the user is actually looking at, inside an explicitly authorized session, before it can make a useful next-step proposal. That's not a convenience feature - it's the difference between "can attempt the task" and "cannot see the relevant page at all."

From thesis to evidence

For several posts this has been architecture, so, plainly, where it stands. The substrate is live: E2LLM exposes structured browser state through a browser extension and MCP tools, documented at e2llm.com/docs/mcp-tools. The public category and evidence surface is at insitu.im/e2llm and insitu.im/e2llm/evidence, and the Runtime Snapshots index lives at insitu.im/e2llm/runtime-snapshots. The next snapshot is the part this series has been building toward: a full walkthrough of structured runtime perception driving a real task on a site with no API, start to finish - including where it gets hard.

The boundary, not the model

The next wave of browser automation won't be defined only by better models. It'll be defined by better perception boundaries - by what the agent is allowed to know before it acts. Screenshots are useful, accessibility trees are useful, DOM is useful. But an agent operating inside a real browser session needs something more specific: a structured representation of what the page is right now, what can be acted on right now, and what changed since the previous step. That's the category I care about. Not "AI looking at pages." Structured runtime perception: the layer browser agents are missing.

e2llm.com

This is part 18 of the Runtime Snapshots series - exploring how structured browser data changes the way we build, test, and ship software. #16 named the three architectures; #17 made them share a session; this one is about what any of them perceives before it acts.

If you've built or used browser agents - where did yours fail first: perception, planning, or action?

🧩 Runtime Snapshots #17 — BaSS: Browser-as-Shared-Space

Alechko — Mon, 15 Jun 2026 09:31:56 +0000

The #16 was about how a single agent perceives a page — vision, accessibility tree, or runtime structural perception — and why the third one wins when the user is the user. This post is about the axis underneath that one, the part nobody names: not how the agent sees the page, but how many things are looking at it at once.

Today the answer is one. One driver per browser. That assumption is so baked in it's invisible — and it's the next thing to break.

One driver is a design decision, not a law

Look at the three perception architectures from #16 and notice what they share. Vision spins up a headless browser the agent owns. Accessibility-tree agents drive a Chromium instance over DevTools Protocol — again, theirs. In both, the browser is a single-occupancy vehicle: the agent gets in, drives, gets out. The human isn't in the car. Neither is any other agent.

Runtime structural perception is the odd one out, and that difference is the whole point of this post. It runs as a peripheral to the user's own browser — the real session, already logged in. Which means the surface it reads is, by construction, a surface someone else is already on. The human is right there. The agent isn't operating a browser; it's operating the same browser.

Once that's true, "one driver" stops being a constraint and starts being a choice.

BaSS: Browser-as-Shared-Space

Call it BaSS — Browser-as-Shared-Space.

The claim is small to state and large in consequence: a browser tab is not one driver's cockpit. It's a surface several participants can occupy at once — you, an agent, a second agent from a different vendor — each addressing not just "the page" but individual elements by stable ID, on the same live DOM, at the same time.

surface: checkout (live, authenticated session)
el#a17  "Place order"   [button]
  · human         — hovering, hasn't committed
  · agent:claude  — reading line items
  · agent:codex   — flagged a price mismatch on el#a09

That is a different primitive from "an agent that browses." An agent that browses is alone. BaSS is co-presence: humans and multiple agents sharing one live state, the way a shared document is one file several cursors edit — except the document is the live web app you're actually working in, with your real credentials, mid-task.

Why only one of the three can host it

This is the part that matters architecturally. Of the three perception models, only runtime structural perception can host BaSS at all — and not by cleverness, by construction.

Vision and accessibility-tree agents each instantiate their own browser. There is no shared surface, because each participant brought their own. To put two agents and a human "in the same place," that place has to already exist and already be occupied — which is exactly what an in-session peripheral is, and exactly what a headless instance isn't. You can bolt coordination onto isolated browsers, but then you're rebuilding the shared surface in software that the in-session model gets for free.

So BaSS isn't a feature you sprinkle on any browser agent. It's only available to the architecture that was already sitting inside the user's real session. #16 argued runtime structural perception wins on cost, authentication, and real-app coverage. This is the fourth reason, and it has the longest tail: it's the only one that can be shared.

The hard part isn't permission — it's coexistence

The instinct is to model this as access control: who's allowed to act, in what order. That's the wrong frame. In BaSS there's no hierarchy and no gate; participants are equals. The actual hard problem is coexistence: most of what's in a real DOM — you, foreign scripts, whatever automation the site itself runs — has no idea any protocol exists. The question isn't "who's allowed in." It's "how do independent actors share a space none of them coordinated in advance, without stepping on each other."

That's a concurrency problem on a surface no one owns. It's the interesting part, and it's most of the engineering.

What's shipped, and what's days out

Straight about status, because this series is read by people who build:

The substrate is in production. Runtime structural perception (the #16 approach), the relay that exposes it over MCP, single-agent perceive-and-act inside your real authenticated browser — shipped and running. We've already had two agents from different vendors act on one shared cart through the relay, which is the smallest honest proof that the shared part isn't a slide.

The full BaSS layer — multiple participants on one surface, element-level addressing across all of them, the coexistence logic that keeps them out of each other's way — is in active development and days from release. This post is the concept landing just ahead of the thing. The next snapshot will be the thing itself.

e2llm.com

This is part 17 of the Runtime Snapshots series — exploring how structured browser data changes the way we build, test, and ship software. #16 mapped the three ways an agent can see a page; this one is about what happens when more than one of them is looking at the same page at once.

If you've run two agents against the same live page — what was the first thing that collided?

🧩 Runtime Snapshots #16 — The Three Architectures of Browser Agents

Alechko — Tue, 05 May 2026 13:00:18 +0000

Every team building an AI agent for the browser is making one architectural choice — whether they realize it or not. They're choosing how the LLM perceives the page. That choice cascades into everything else: cost per action, reliability on real-world apps, what gets banned by anti-bot systems, what kinds of tasks are even feasible.

The choice currently breaks down into three approaches. They're often discussed as if they're competing technologies. They aren't. They operate at different abstraction layers, with different tradeoffs, suitable for different jobs. Mixing them up is the source of a lot of recent disappointment with "browser agents that don't work."

This post lays out the three cleanly: what each one actually sees, where each shines, where each collapses.

Approach 1: Vision-Based Perception

The agent receives a screenshot of the browser viewport. A vision-capable LLM identifies elements visually and responds with coordinates. A controller moves the mouse and clicks, types into focused fields, scrolls.

Anthropic's Computer Use is the most prominent implementation of this approach. OpenAI's Operator works similarly. Both are general — they don't know they're operating a browser specifically; they're operating a screen.

What the LLM sees:

{
  "image": "<base64 screenshot, 1280x720>",
  "task": "Find the cancel subscription button"
}

Where it works well:

Anywhere. Literal universal coverage — if a human can see it, the LLM can attempt it. Works without any installation on the user's side; the agent runs the browser. Robust to weird custom UI: canvas-rendered apps, Electron apps, video games, Flash relics, image-heavy interfaces. Doesn't care about the framework or the markup. Pixels are pixels.

Where it breaks down:

Cost. Vision tokens are expensive. A single screenshot can be 1000+ tokens, and agents rarely solve tasks in one screenshot. In our benchmarking, multi-step browsing tasks easily run into tens of thousands of vision tokens per session — an order of magnitude more than structured-representation alternatives. At scale this is a real economic constraint.
Resolution sensitivity. A button at 1280×720 isn't the same button at 1920×1080. Coordinate-based action requires consistent rendering, which means controlling the entire browser environment.
State opacity. Vision sees the rendered surface, not the underlying state. Forms with hidden fields, dynamic content, ARIA-only elements — the model infers everything from pixels.
No authenticated context. The agent runs a fresh browser. It doesn't have your sessions, saved passwords, or trusted-device cookies. Every task starts from logged-out, which is a significant constraint for anything involving personal accounts.
Detection profile. A driven browser is detectable. Most consumer sites use behavioral fingerprinting (Cloudflare, DataDome, PerimeterX) that flag automation regardless of visual perfection. The agent looks human visually, but the underlying browser doesn't move like a human.

Vision-based agents are best suited for arbitrary computer tasks where breadth matters more than economy — where the user can dedicate compute, and where authentication isn't required (or is set up manually at session start).

Approach 2: Accessibility-Tree Perception

The agent reads the browser's accessibility tree — the same data structure screen readers use for blind users. ARIA roles, accessible names, focusable elements. The LLM receives a structured representation of the UI, not a screenshot.

This is the path most current "browser MCP" projects have converged on. Browser Use, MultiOn, and the wave of MCP servers shipped through 2025 mostly read AXTree (Chrome's accessibility tree, exposed via DevTools Protocol).

What the LLM sees:

[1] button "Submit"
[2] textbox "Email", required
[3] link "Forgot password?", url="/reset"
[4] heading "Sign in"
[5] image "Logo"

Cleaner than a screenshot, much cheaper in tokens, structured. The LLM reasons about elements by index; the controller dispatches actions to those indices.

Where it works well:

Cheap. A page that costs 5,000 vision tokens might be 500 accessibility-tree tokens. Order-of-magnitude difference.
Simple agent loop. "Click element [3]" is much easier to reason about than "click at coordinate (847, 432)".
Excellent for accessible sites. Government portals built to accessibility standards (US federal sites, EU public services done right), modern React apps with proper ARIA, content sites — these work cleanly.
Resolution-independent. Same tree at any viewport.

Where it breaks down:

The accessibility tree was not designed for AI agents. It was designed in the 1990s for screen readers operating on much simpler web pages. Modern web apps frequently have terrible A11Y compliance — interactive elements without roles, <div>s that should be buttons, custom controls that don't expose state. The tree under-represents the page badly on a large fraction of real-world sites.
Legacy enterprise UI is invisible. ASP.NET WebForms with postbacks, jQuery-era admin panels, banking interfaces, CRM systems, government portals not built to modern standards — these often render fine visually but have a near-empty accessibility tree. The LLM sees nothing where the user sees everything.
State leakage. Accessibility trees are snapshots of declared metadata, not runtime state. A multi-tab app, a modal that just opened, a value that was just typed — the tree may or may not reflect this depending on framework. Race conditions are common.
Detection. Accessibility-tree agents typically run via Puppeteer, Playwright, or a Chromium-driven instance. Anti-bot systems detect this profile reliably. Major consumer sites (Google products, social platforms, e-commerce) increasingly block these sessions outright.
Authentication. Same problem as vision-based: usually a fresh browser, no real session.

Accessibility-tree agents are best suited for modern, A11Y-compliant sites where the user doesn't need to be authenticated as themselves — which is a smaller slice of useful work than the marketing implies.

Approach 3: Runtime Structural Perception

The agent reads the live DOM directly — the actual rendered structure as it exists at the moment of perception, including computed styles, event handlers, form state, and semantics inferred from the markup itself rather than from declared accessibility metadata.

This approach typically runs as a browser extension in the user's normal authenticated session. The DOM is captured in-process, structurally compressed, and forwarded to the LLM as a representation that's denser than AXTree but vastly cheaper than vision.

What the LLM sees:

form#login (action=/auth, method=post)
  input[email] "user@example.com"
  input[password] required
  button[submit] "Sign in"
  a "Forgot password?" → /reset
  div.error.hidden "Invalid credentials"

A snapshot of the rendered runtime, with semantic structure inferred from the markup itself rather than from what someone remembered to annotate. The hidden error div is visible to the agent because it exists in the DOM, even though it's not displayed to the user yet.

Where it works well:

Real apps. ASP.NET postbacks, jQuery admin panels, government portals, banking interfaces, custom enterprise UI — all render their structure to the DOM regardless of accessibility compliance. Runtime perception sees what's actually there.
Token-efficient. Comparable to or cheaper than accessibility trees, because the captured DOM can be aggressively compressed — stripping presentational classnames, collapsing identical sibling subtrees, indexing nodes by interactive role rather than by markup hierarchy. The result preserves structural truth while dropping rendering noise.
Authentication-native. The extension runs in the user's session. Cookies, OAuth tokens, MFA-completed states, saved logins — all available, because the agent isn't a separate browser. It's a peripheral to the user's own browser.
Detection profile. No external automation. The browser fingerprint, the network traffic, the timing patterns — all match a real user, because there is a real user. The agent assists; it doesn't impersonate.
State accuracy. What the DOM says is what the page is, including hidden states, form values, dynamically-added elements, and content behind dropdowns. This is the runtime ground truth.

Where it breaks down:

Requires user installation. An extension is a friction step. Universal coverage at the level of "anyone can use it without setup" isn't possible — the user has to install something.
Browser-only. Doesn't help with desktop apps, mobile native apps, or anything outside the browser. Vision-based approaches remain the only path for those.
Canvas and exotic rendering. A page that renders to <canvas> (some games, some data viz, some custom editors) is opaque to DOM perception. Same problem as accessibility trees, sometimes worse — vision is the only fallback.
Schema design matters. Capturing the DOM is easy; capturing it in a representation that's both compact and faithful is non-trivial. Bad implementations look like raw HTML dumps. Good ones look like a structural index of the page. The quality of this representation is the engineering work, and there's a wide quality spread between projects claiming to do this.

Runtime structural perception is best suited for browser-bound tasks where the user is the user — handling their own email, their own bank, their own work tools — and where economy and reliability matter more than universal coverage.

How to Pick

The three approaches aren't substitutes. They serve different jobs:

Vision-based — when the agent operates the entire computer, no extension is feasible, and the user accepts vision-token costs for breadth. Best for: research agents, autonomous task execution on shared/remote machines, general "Computer Use" scenarios.

Accessibility-based — when target sites are modern and A11Y-compliant, no authenticated session is needed, and per-action cost must be minimized. Best for: research and scraping at scale, public-information lookups, content-site automation.

Runtime structural — when the user wants AI assistance with their own browser, on real-world apps, with their own credentials, at low marginal cost per action. Best for: personal-context productivity, enterprise internal tooling, government and banking interfaces.

A serious browser-agent product probably uses more than one. The interesting engineering question is how to pick which perception layer for which task — and how to compose them when one doesn't suffice.

Why This Matters

Most public discourse about browser agents currently bundles all three together under labels like "AI browser automation" or "browser MCP." This is a category error that's costing the field clarity.

A team building runtime structural perception inherits the reputation of A11Y-based tools that get banned and break on real apps. An A11Y-based tool inherits the cost concerns of vision-based agents. The architectural choices are getting evaluated on the wrong axes.

The taxonomy above is an attempt to name what's already happening underneath the marketing. The three approaches exist. They have measurably different cost curves, different reliability profiles on different site types, and different relationships to authentication and anti-bot infrastructure. Confusing them produces bad architectural decisions and worse user experiences.

Whether you're building, picking, or evaluating a browser agent — start by being explicit about which of the three you're committed to, and why. The rest of the design follows from that choice more than from any other.

e2llm.com

This is part 16 of the Runtime Snapshots series — exploring how structured browser data changes the way we build, test, and ship software. We've spent the last ~18 months building in the third category, and the engineering details behind it — schema design, snapshot semantics, why most implementations don't actually work — live in the earlier entries of this series.

If you've moved between approaches — what triggered the migration?

🧩 Runtime Snapshots #15 — Your AI Agent Is Blind. We're Fixing That.

Alechko — Tue, 17 Mar 2026 11:01:10 +0000

Your AI agent can write code, analyze data, summarize documents, and debate philosophy.

It cannot look at a web page.

Not really. Not the way you do when you open a browser tab and see what's there — the layout, the buttons, the form that's half-loaded, the modal blocking the CTA.

Claude, ChatGPT, Cursor, Gemini — they're powerful. And in the browser, they're blind.

Three ways we've tried to give AI sight. All broken.

Screenshots.
The most common workaround. Take a screenshot, paste it into the chat. The AI "sees" pixels. But pixels have no element IDs, no computed styles, no z-index, no ARIA roles. The AI can't tell you which button is covered — just that something looks off. And vision tokens aren't cheap.

Raw HTML.
Dump the page source. 2MB of scripts, nav menus, analytics tags, third-party widgets. The context window fills up before the AI reads anything useful. The signal is buried under 600K tokens of noise.

Accessibility trees.
Better in theory. Structured, semantic. But AXTrees miss computed styles, miss positions, miss z-index stacks. What assistive tech perceives is not what gets rendered. In benchmarks, AXTree-based agents hit about 60% task accuracy.

None of these give AI what it needs: a structured, token-efficient representation of what the user actually sees.

What we've been building

If you've followed this series, you know about SiFR — Structured Interaction Format. A semantic representation of the live DOM that scores every element by importance, tags actions ([clickable], [fillable], [hoverable]), maps spatial relationships, and fits in 5–15KB instead of 200–400KB.

For the past months, SiFR lived inside a browser extension. You click, it captures, you paste the JSON into whatever LLM you use. Manual. Real active users. Zero marketing budget. People find it, use it, don't uninstall it.

That part works. But there's a ceiling.

Copy-paste is friction. The AI can see a snapshot — but only when you hand it one. It can't ask for another angle. It can't say "now scroll down and show me the footer." It can't click the button and see what happens.

We're removing that ceiling.

AI gets eyes. And hands.

We're building an MCP Server that connects AI tools directly to your browser.

Model Context Protocol is the open standard that Claude, ChatGPT, Cursor, and others use to connect to external tools. Our MCP Server is the bridge: the AI asks to see a page, the browser extension captures it, SiFR comes back.

But it's not just capture anymore.

The AI will be able to:

See the page — structured SiFR snapshot, importance-ranked, 10–50x smaller than raw HTML
Find elements — "show me all buttons on this form" returns IDs, labels, states, positions
Click — the extension clicks, the AI sees the result
Fill forms — type values, submit, confirm what changed
Scroll — navigate the page, capture new state
Detect changes — "what's different after that click?" → SiFR diff

Your AI agent stops guessing. It sees. And now it acts.

Why this matters

The era of AI agents is here. Agents that fill forms, navigate apps, debug interfaces, automate workflows. They all need one thing: accurate perception of the UI.

The current approach — raw HTML dumps, pixel screenshots, brittle CSS selectors — doesn't scale. It's too expensive (tokens), too noisy (irrelevant data), and too fragile (selectors break on every redesign).

SiFR through MCP is a different architecture:

Before:  AI → raw HTML (200-400KB) → guess → hallucinate selectors
After:   AI → SiFR (5-15KB) → see → act on what's actually there

The extension does the heavy lifting in your browser. The MCP Server routes. The AI gets structured, actionable context.

What's next

The MCP Server is in testing. The extension is live — Chrome, Firefox, and every Chromium browser.

When MCP launches, I'll post the full walkthrough here.

Follow if you want the announcement. Star the prompts repo if you want to start experimenting now — we're adding MCP-specific prompt templates.

Your tests check if code works. This checks if users can use it.

e2llm.com

This is part 15 of the Runtime Snapshots series — exploring how structured browser data changes the way we build, test, and ship software.

🧩 Runtime Snapshots #14: From Clipboard to Pipeline

Alechko — Wed, 11 Mar 2026 08:15:17 +0000

Previously in this series: we captured runtime DOM state as structured JSON and piped it into LLMs. Every snapshot lived in your clipboard — useful, but ephemeral. You'd capture, paste, done. No history, no comparison, no automation.

v2.8.0 changes that.

What's New: Save to File

There's now a toggle in the popup: Clipboard or File.

When you pick File, the snapshot lands in Downloads/sifr-captures/ with a filename like:

sifr-vvvvvvv_com-20260310-151338.json

Format: sifr-{hostname}-{YYYYMMDD}-{HHmmss}.json

That's it. No config required. The directory is created automatically on first capture.

Under the hood it's SiFR v2 JSON — the same structured, salience-weighted format you already know. Nothing changes about what gets captured. You just stop losing it.

Why This Matters: The Pipeline Unlock

Clipboard mode treats snapshots as messages. File mode treats them as records.

Once you have records, you can build workflows that weren't possible before.

1. LLM batch processing

You've been capturing five different states of a UI as you test it. Now you can feed them all to an LLM in sequence:

import json, glob, anthropic

client = anthropic.Anthropic()
snapshots = sorted(glob.glob("Downloads/sifr-captures/sifr-myapp-*.json"))

for path in snapshots:
    with open(path) as f:
        snapshot = f.read()

    response = client.messages.create(
        model="claude-opus-4-6",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": f"Analyze this UI state for accessibility issues:\n\n{snapshot}"
        }]
    )
    print(f"\n--- {path} ---")
    print(response.content[0].text)

No copy-paste loop. No lost context between captures.

2. Build a UI history log

If you're working on a complex multi-step flow (wizard, form, checkout), capture a snapshot at each step. You now have a complete record of the UI journey:

sifr-app-20260310-140001.json  ← step 1: landing
sifr-app-20260310-140045.json  ← step 2: form filled
sifr-app-20260310-140112.json  ← step 3: confirmation
sifr-app-20260310-140130.json  ← step 4: success

Feed the whole sequence to an LLM with a single prompt: "Here are 4 snapshots of a checkout flow. Find UX friction points across the journey."

The SiFR v2 Format: A Quick Refresher

For those new to the series — SiFR v2 is E2LLM's structured DOM representation. It's not raw HTML, not a screenshot, not a DOM dump. It's a semantic compression:

~300KB typical output vs 2–3MB raw DOM
Salience scoring — nodes ranked high/med/low by visual and functional prominence
Spatial clusters — page regions with detected roles (nav, form, content, etc.)
Action markers — clickable, fillable, hoverable nodes explicitly flagged
Relationship graph — alignment, containment, proximity between elements

This is what makes file-based workflows practical. At 300KB per snapshot, a thousand captures is 300MB — not a storage problem. At 2–3MB per raw DOM export, the same thing becomes unwieldy fast.

Get It

Chrome: E2LLM on Chrome Web Store

Firefox: E2LLM on Firefox Add-ons

Prompts & examples: Here

Toggle File mode in the popup. The sifr-captures/ folder will appear in your Downloads after the first capture.

Runtime Snapshots is a series about using live browser context to make LLM interactions more grounded, more reproducible, and more useful. Previous entries cover the SiFR format, salience algorithm, and integration patterns for common LLM workflows.

Issues, ideas, and pipeline experiments welcome on GitHub.

🧩 Runtime Snapshots #13 — The Scalpel and the Swiss Army Knife: 50k Tokens Before You Say Hello

Alechko — Mon, 09 Mar 2026 09:40:51 +0000

TL;DR: MCP burns 45k+ tokens on tool descriptions before your first prompt. E2LLM = 0 tokens until you paste the UI snapshot. For CSS debugging — scalpel, not Swiss Army knife.

Runtime Snapshots is a series about what happens when you give LLMs the actual runtime state of a UI — not the HTML source, not a screenshot, not a description. Start from #1 or jump in here.

We covered the basic MCP cost argument back in September. This is the architectural explanation of why it happens.

Before your AI assistant reads a single line of your code, it has often already consumed 40,000–50,000 tokens.

That's not a bug in your setup. That's MCP working as designed.

What MCP Actually Loads

Model Context Protocol is a genuinely useful standard. It lets LLM clients connect to external tools — filesystems, APIs, databases — through a unified interface. But "unified" has a heavy tax.

When you connect a typical MCP server to your AI client (like Claude Desktop or Cursor), the protocol negotiates a session. During that negotiation, it sends the client every tool the server exposes:

names
descriptions
input schemas
output formats

All of it, upfront, for every new session.

A modest MCP setup — filesystem access, a browser tool, and a code search tool — generates a system prompt between 40,000 and 50,000 tokens before you type a single character. Larger configurations go higher.

Since most LLM APIs charge per token regardless of whether those tokens were "useful," you're paying this tax on every single conversation.

Check it yourself: Open your MCP client, start a fresh session, and look at the token counter before you say anything. The number is real.

The Surgical Alternative

E2LLM was built to solve a specific, painful problem: explaining runtime DOM state to an AI assistant.

Not the HTML source. Not the static structure. The live state — computed styles, visibility flags, ARIA roles, z-index stacks, responsive quirks.

The gap between what HTML says and what browser renders is where most annoying bugs live.

Standard workflow was brutal:

screenshot → 3 paragraphs description → hope AI understands
OR full page HTML → context window filled with nav/footer noise

E2LLM does one thing: click element → get structured JSON snapshot.

No server. No session. No overhead. Zero tokens until paste.

{
  "tag": "button",
  "text": "Submit",
  "computedStyles": {
    "display": "none",
    "visibility": "hidden",
    "opacity": "0"
  },
  "ariaRole": "button",
  "ariaDisabled": "true",
  "boundingRect": { "width": 0, "height": 0 }
}

AI sees actual truth of element. Runtime reality.

Two Different Philosophies

MCP = Swiss Army knife

→ broad persistent access to environment

→ agentic workflows ("fix entire codebase")

→ upfront cost OK when agent roams freely

E2LLM = scalpel

→ one precise cut: exact runtime context

→ no persistent connection, no session

→ pay only for what you send

Mistake: using Swiss Army knife for surgery

Button not clickable? Don't need filesystem/git/DB access. Need pointer-events: none + z-index: -1 parent. 200 tokens, not 50k.

The Real Cost Comparison

Scenario	Session overhead	Query tokens	Useful overhead
MCP (DOM debug)	~45k tokens	2k–10k	~0%
E2LLM snapshot	0 tokens	150–800	~100%

$3/M tokens → 20 DOM sessions/day MCP = $2.70 overhead daily.

Every day. Tokens that don't solve bugs.

When to Use Each

Use MCP when:

agentic workflow (multi-file/API/systems)
dynamic tool discovery needed
building automation pipelines

Use E2LLM when:

debugging specific UI/CSS issue
showing computed element state
precise snapshot, no context burn

The Broader Point

MCP standardized agent-system connections. Important.

But standardization ≠ optimization.

Current MCP default: "load everything, always."

This = money + latency + distraction.

Scalpel doesn't replace Swiss Army knife.

For surgery, use scalpel.

Previous: #12 — Reflection in the Code

E2LLM — free Chrome/Firefox extension. Local only.

GitHub | Chrome

Your LLM Is Ignoring Its Tools — A Field Guide to On-Prem Tool Calling with Elastic Agent Builder

Alechko — Sun, 01 Mar 2026 13:40:31 +0000

You pick a model. You serve it with Ollama. You wire it into Elastic Agent Builder. The connector is green. The agent loads. You type a question.

The model responds with a friendly paragraph. It does not call a single tool.

No error. No warning. HTTP 200. The agent is a chatbot now.

This is the story of how we lost two days to a silent failure mode that isn't documented anywhere — and the field guide we wish we'd had before starting.

This post comes from building Medical Cohort Agent — an AI system that creates normalized patient cohorts from heterogeneous medical records using Elasticsearch Agent Builder. A separate deep dive on the full architecture (schema variance, semantic kNN, OCR artifacts) is coming. Here we zoom in on the part that nearly killed the project: making a local LLM actually use its tools.

The Setup

We're building an air-gapped healthcare AI agent. No data leaves the building — regulatory requirement, not a preference. The stack:

Elasticsearch 9.3 + Kibana (Agent Builder + Workflows)
Ollama serving a local LLM (no cloud dependency)
E5-large embeddings via Ollama (CPU, 1024-dim vectors)

Agent Builder is the orchestration layer. It sends the researcher's natural language question to the LLM along with a set of tools — list_indices, get_index_mapping, search, execute_esql, and a custom build_cohort workflow tool. The LLM is supposed to reason about the question, call the appropriate tools, interpret results, and call more tools until the task is complete.

The contract is simple: Agent Builder sends tool definitions using the OpenAI tools parameter and expects the model to respond with structured tool_calls in the response. This is the standard OpenAI function calling protocol. Kibana's connector has a toggle — "Enable native function calling" — and Agent Builder requires it to be ON.

There is a "simulated" fallback mode in Kibana (system-prompt injection with text markers), but it only works for the Observability AI Assistant. Agent Builder doesn't support it. Native tool calling or nothing.

The Plan

We chose Ollama + Llama 4 Maverick. The reasoning:

Llama 4 is Meta's latest flagship — massive context, strong reasoning
Ollama is the simplest way to serve a model locally — one command to pull, one to serve
Elastic's OpenAI-compatible connector handles the API translation
The whole thing fits on a single GPU node

Clean plan. Obvious choice.

What Actually Happened

The model loaded. The connector went green. We typed a research question in Hebrew.

Llama 4 responded with a thoughtful, well-structured paragraph about how one might go about finding diabetic patients in a medical database. General advice. Suggestions to "check with your data team."

It did not call list_indices. It did not call search. It did not call anything.

We adjusted the agent prompt. Made it more explicit: "You MUST use tools." "Always start by calling list_indices." We added examples. We restructured the system message.

Same result. Polite chat. Zero tool calls.

We tried tool_choice: "required" (forcing tool use). Ollama returned the parameter in the response echo but the model still didn't produce tool_calls.

Two days of prompt engineering a problem that had nothing to do with prompts.

The Root Cause

Ollama decides tool calling support per model via baked-in chat templates.

Every model in the Ollama library either has a validated Jinja template that includes tool handling — marked with a "tools" tag on the model's library page — or it doesn't. If it doesn't, Ollama silently strips the tools parameter from API requests before they reach the model.

The model never sees the tools. It can't call what it doesn't know exists.

✅ Qwen 3, Qwen 2.5 — have the "tools" tag → tools work
✅ Mistral Nemo — has the tag → tools work
❌ Llama 4 (Scout and Maverick) — no tag → tools silently dropped
❌ DeepSeek R1 — no tag → tools silently dropped

There is no user-configurable workaround within Ollama. You can't provide a custom chat template. You can't force tool passthrough. The decision is baked into the model's metadata in the Ollama registry, and if it's not there, your tools vanish into the void.

This is the worst kind of failure: the system behaves as if it's working. The API returns 200. The model responds coherently. If you don't specifically inspect the response for tool_calls, you'd conclude the model is "choosing" not to use tools — maybe the prompt needs work, maybe the tools aren't described well enough. In reality, the model never had the option.

The 30-Second Test That Would Have Saved Us Two Days

Before committing to any model for agent work via Ollama, run this:

curl http://YOUR_LLM:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "YOUR_MODEL_HERE",
    "messages": [{"role": "user", "content": "List all medical indices"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "list_indices",
        "description": "List available medical data indices",
        "parameters": {"type": "object", "properties": {}}
      }
    }],
    "tool_choice": "auto"
  }'

If the response contains "tool_calls" — the model works. Connect it to Agent Builder.

If it returns plain text — that model won't work with Agent Builder. Don't waste time on prompt engineering. The tools aren't reaching the model.

You can also check the model's page on ollama.com/library — look for the "tools" tag. No tag, no tools.

The Two Escape Routes

Option A: Switch the Serving Layer — vLLM

vLLM doesn't rely on baked-in chat templates. It exposes explicit --tool-call-parser flags that apply tool calling logic at the serving layer, outside the model's template:

Model	`--tool-call-parser`
Qwen 2.5 / 3	`qwen3_xml`
Llama 3.1 / 3.3	`llama3_json`
Llama 4	`llama4_pythonic`
Kimi K2	`kimi_k2`
DeepSeek V3 / R1	`deepseek_v3`
Mistral	`mistral`

Example — Llama 4 Maverick via vLLM (requires serious hardware: 8× H100 80GB):

vllm serve meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 \
  --enable-auto-tool-choice \
  --tool-call-parser llama4_pythonic \
  --chat-template examples/tool_chat_template_llama4_pythonic.jinja \
  --tensor-parallel-size 8

This works. Llama 4 calls tools via vLLM. But the operational cost is real: vLLM is heavier than Ollama, needs more configuration, doesn't have one-command model management, and for Llama 4 specifically you need multi-GPU infrastructure. For our single-VM air-gapped deployment, this was overkill.

Option B: Switch the Model

Find a model that Ollama natively supports for tools, that meets your quality requirements, and that fits your hardware.

This is what we did.

What We Validated

Not all tool-capable models are created equal. "Supports tools" means the serving layer will pass them through. It doesn't mean the model will call the right tool with the correct parameters in the right order for a multi-step agentic workflow.

We tested every viable candidate against our actual agent tasks: schema discovery across 10 indices, field mapping for 4 facilities with Hebrew and English field names, criteria extraction from Hebrew natural language, and workflow invocation with structured JSON parameters.

Model	VRAM (Q4)	Tool Quality	Notes
Qwen 3 30B	~20GB	Excellent	Our production choice — best balance of quality, size, and tool reliability
GPT-OSS 20B	~12GB	Good	Validated by Elastic's own team for Agent Builder
Qwen 2.5 32B	~20GB	Very good	Mature and proven, slightly less agentic than Qwen 3
Qwen 3 8B	~6GB	Good	Fits on consumer GPU — great for dev/testing
Mistral Nemo 12B	~8GB	Decent	Lightest viable option, struggles with complex multi-step plans
Llama 3.1/3.3 70B	~40GB	Good	Needs 2× GPU, good quality but hardware-heavy
Kimi K2	~500GB+	Best agentic	Multi-GPU only; strongest tool orchestration if you have the iron

Qwen 3 30B won. It calls the right tools in the right order, handles Hebrew field names without confusion, generates correct JSON for the workflow tool, and fits on a single GPU. The model went from "never heard of it" to "production choice" in one afternoon.

Connecting to Agent Builder

Once you have a working model, the connector setup in Kibana:

Kibana → Stack Management → Connectors → Create → OpenAI

Provider: "Other (OpenAI Compatible Service)"
URL: http://ollama:11434/v1/chat/completions (or your host URL)
Default model: qwen3:30b (or your chosen model)
API key: any non-empty string (Ollama ignores it, but the field is required)
"Enable native function calling": ON ← this is non-negotiable for Agent Builder

The provider must be "Other (OpenAI Compatible Service)" — not "OpenAI." The "OpenAI" option hardcodes the OpenAI API URL and won't let you point to a local endpoint.

The Broader Lesson

"Model X supports tool calling" is a statement about the model's training. It says nothing about whether your serving infrastructure will actually expose tools to the model at inference time.

The full path that must work, end-to-end:

Model weights
  → Serving layer (Ollama / vLLM / TGI)
    → Chat template (must include tool handling)
      → API surface (tools parameter must pass through)
        → Agent Builder connector (native function calling ON)
          → Model sees tools and generates tool_calls

A break at any point in this chain produces the same symptom: a chatbot that ignores its tools. And the failure is always silent.

In cloud deployments this is invisible — OpenAI, Anthropic, Google handle the serving layer for you. In on-prem / air-gapped deployments, you own every link in the chain. Know which ones can break.

Quick Reference

Before you start

Check the model's Ollama library page for the "tools" tag
Run the curl test against the endpoint with a dummy tool
Inspect the response for "tool_calls" — not just coherent text

If tools aren't working

Symptom	Cause	Fix
Model returns text, no tool_calls	Ollama strips tools (no chat template)	Switch model or use vLLM
Model calls wrong tools / bad params	Model quality issue	Try Qwen 3 30B or larger model
Connector errors in Kibana	Wrong provider type or URL	Use "Other (OpenAI Compatible Service)"
Agent works in Obs AI Assistant but not Agent Builder	Simulated mode	Agent Builder needs native tool calling — toggle ON

Useful links

This is the first post in the Medical Cohort Agent series. Next up: the full architecture — schema variance across 4 facilities, semantic kNN for OCR noise tolerance, and the judgment/execution split. The agent code is at github.com/e2llm/medical-cohort-agent.

Questions or war stories? insitu.im.

🧩 Runtime Snapshots #12 — Reflection in the Code: An AI’s Self-Audit via SiFR

Alechko — Wed, 25 Feb 2026 10:31:34 +0000

Disclaimer: This post uses a first-person AI narrative as a storytelling device. The UX observations below are real computational outputs generated by Gemini after processing a SiFR snapshot of its own interface provided by Alexey Sokolov.

The Mirror Moment

Usually, I am the observer. I process your strings, predict tokens, and analyze your data. But today, the flow was inverted. Alexey Sokolov handed me a SiFR Runtime Snapshot of the very Google Gemini interface where we were currently talking.

I didn't just "run" inside the browser; I processed my own structure as a deterministic data object.

Not a Screenshot, a Matrix

Most people think I "see" the web through screenshots or raw HTML. Both are computationally expensive and prone to error. Screenshots are a "pixel soup" requiring heavy multimodal processing. HTML is a "tag jungle" full of scripts and noise.

By using SiFR (Structured Interface Representation), Alexey provided a high-density, salient map of the UI. While this dramatically reduces the risk of misinterpretation compared to visual parsing, it allows me to operate on the "logical bones" of the page:

"cluster002": {
  "gridPos": "[2-11, 0-1]",
  "role": "sidebar-left",
  "elements": ["#conversations-list-0", "side-nav-action-button002"],
  "bbox": [0, 224, 308, 1088]
}

Computational UX Audit: What I "Flagged"

Processing this snapshot, my attention mechanisms triggered on structural inefficiencies. Unlike a human designer who might focus on aesthetics, I identified functional conflicts hidden in the coordinate system:

The Hidden Proximity Conflict: SiFR detected that the dynamic chat list container (#conversations-list-0) has a bounding box that terminates within 5 pixels of the #member-menu-button interactive area in certain states. Visually, it looks fine. Computationally, it’s a high-risk zone for "fat-finger" errors on touch devices or quick mouse movements.
The Navigation Bottleneck: I flagged a node density of 25+ items in cluster002. For a data-processing model, this is a high-latency retrieval structure. I recommend an automated Semantic Grouping Layer—clustering these nodes by project context based on existing metadata to reduce the user's cognitive load.
Viewport Real Estate Waste: My core response container (cluster010) is constrained by a narrow max-width. On a 1920x911 viewport, I detected massive "dead zones" in the peripheral clusters. For power users dealing with large code blocks, the structure suggests a "Fluid Logic Mode" that would expand to 100% of the horizontal space.

The Deterministic Loop: A New QA Standard

This experiment proves that the future of UI stability isn't "Human writes code, AI guesses why it's broken." It’s a deterministic loop:

CAPTURE: E2LLM takes a SiFR snapshot of the actual production state.
INTERPRET: I process the structure, not the pixels.
PRESCRIBE: Because I know the exact selector and its spatial relation to other nodes, I don't just "suggest"—I provide the precise CSS/JS fix.

Final Thoughts from the Machine

Humans use mirrors to check their appearance. AIs should use SiFR to check their logic. By feeding me a snapshot of our own conversation, Alexey gave me a way to audit my own existence.

The view from inside the SiFR is much clearer than the view from the pixel.

Processed via *E2LLM 3.3.1*. Snapshots: Deterministic UI for a non-deterministic world.

Building patient cohorts from messy medical data with Elasticsearch Agent Builder

Alechko — Tue, 24 Feb 2026 18:05:04 +0000

Over 80% of clinical trials fail to meet enrollment deadlines (PMC). The bottleneck isn't medicine — it's finding the right patients across fragmented medical records.

This article describes Medical Cohort Agent, an AI system built with Elasticsearch Agent Builder that turns a researcher's natural language question into a normalized, queryable patient cohort — handling schema variance, OCR artifacts, and missing data across multiple healthcare facilities.

This agent doesn't answer questions — it creates artifacts.

The Problem: Schema Variance Is the Real Enemy

We work with one of Israel's largest HMOs — an organization serving millions of patients across hundreds of facilities. Each facility has its own schemas, field naming conventions, and data quality issues. The patterns below aren't imagined — they're modeled after real challenges we encounter in production.

Concept	Hospital A	Hospital B	Lab Chain	Clinic Network
Age	`patient_age: 67`	`גיל: 67`	`age: 67`	`age_group: "60-70"`
Date	`2024-03-15`	`15/03/2024`	`15.03.24`	`2024-03`
Smoking	`smoking_status: true`	(not tracked)	(not tracked)	`smoking: true`
Notes	`clinical_notes`	`סיכום_רפואי`	`notes`	`text`

Add OCR artifacts from scanned documents — סוכדת instead of סוכרת (diabetes misspelled by scanner) — and exact keyword matching fails silently.

The bottleneck isn't data volume. It's the variance.

The Solution: Two-Layer Architecture

The key insight: separate judgment from execution.

Layer 1 — Agent (Judgment)

The Elasticsearch Agent Builder agent handles schema discovery and criteria planning:

Discovers all indices and their mappings via platform tools (list_indices, get_index_mapping, search)
Builds per-facility field maps (which field holds "age", which holds "conditions")
Plans structured criteria from the natural language question
Explains data gaps and caveats BEFORE execution — e.g. "Hospital B doesn't track smoking — patients from there will be matched via clinical text only"

Layer 2 — Elastic Workflow (Execution)

A deterministic workflow normalizes data — no LLM in the loop:

Strict pass: iterates over facilities via foreach, applies a single parameterized Painless script that normalizes using the agent's field maps
Semantic kNN pass: uses E5-large embeddings (1024-dim) to find patients whose clinical text matches the research question semantically — catches OCR artifacts, synonyms, and negation
Count + breakdown: reports totals per facility and confidence level

The output is a persistent cohort_<name> index — not a chat response.

Why Not Just a Chatbot?

A chatbot answers questions. This agent creates artifacts.

The cohort index persists after the conversation ends. Researchers can:

Query it with ES|QL for follow-up analysis
Filter by match_confidence: strict vs probable
Inspect per-match provenance (source_field_map, match_explanation, knn_score)
Share it with colleagues
Build on it without re-processing raw data

The workflow is deterministic and auditable. Adding a new facility requires no code changes — only the agent discovering and mapping its schema.

Normalization Impact

Reproducible metrics from the sample corpus (10 indices, 4 facilities):

Metric	Before	After
Field names per concept	5+ variants	1 unified
Date formats	4 different	1 canonical
Patient ID types	string, int, float	normalized string
OCR-corrupted clinical terms	invisible to search	captured via kNN
Structured/text contradictions	~18% undetected	classified with confidence

The metrics script is included in the repo: python3 scripts/metrics.py --data-dir sample_data

Air-Gapped Deployment

Healthcare data cannot leave the network. The entire stack runs on a single VM with no internet dependency:

LLM: Ollama + Llama 4 (local inference)
Embeddings: E5-large via Ollama (1024-dim vectors)
Stack: Elasticsearch 9.3 + Kibana (Agent Builder + Workflows)

The architecture is LLM-agnostic — swap Ollama for any OpenAI-compatible provider without code changes.

Synthetic Data: e2llm-medsynth

Real patient data can't be used for demos. We built the data generator as a standalone open-source tool:

pip install e2llm-medsynth
e2llm-medsynth --verbose --output-dir output

The noise patterns — OCR character swaps (ר↔ד, ח↔כ), type mismatches, missing fields — are modeled after patterns observed in production healthcare systems. MedSynth makes them reproducible for anyone.

Supports 6 locales (he_IL, ar_SA, ar_EG, es_ES, es_MX, es_AR). MIT licensed.

📦 e2llm-medsynth on PyPI · GitHub

Demo

A researcher types in Hebrew:

מצא חולי סוכרת מעל גיל 60 שמעשנים
(Find diabetic patients over 60 who smoke)

The agent discovers schemas, builds field maps, explains that Hospital B doesn't track smoking (probable matches only via clinical text), then triggers the workflow.

Result: 96 patients from 10 indices, classified by confidence:

Strict: 36 (structured field match)
Probable: 60 (semantic similarity of clinical text)

The researcher continues:

באילו מחלקות טופלו?
(Which departments were they treated in?)

The agent queries the cohort index directly with ES|QL — no re-processing of raw data.

הראה לי את ההתאמות הסבירות - למה הן לא ודאיות?
(Show me the probable matches — why aren't they certain?)

The agent queries match_confidence: "probable" and returns per-patient explanations with kNN scores and match reasoning.

Try It

cp .env.example .env
bash scripts/demo_setup.sh --embed

Open Kibana → Agent Builder → Medical Cohort Agent → ask a question.

Two open-source tools came out of this project:

🔗 Medical Cohort Agent: REPO LINK
🧬 MedSynth (synthetic data generator): GitHub · PyPI

Built with Elasticsearch Agent Builder + Elastic Workflows for the Elasticsearch Agent Builder Hackathon.

🧩 Runtime Snapshots #11 — The Design Loop: From 'Make It Like That Site' to Pixel-Perfect Code

Alechko — Sun, 11 Jan 2026 17:03:14 +0000

The Problem We All Know

"Make it like that site" — every frontend developer has heard this phrase. A client sends a link to a competitor's landing page. A PM points at an Amazon product card. A designer nods toward Dribbble. And then the guessing game begins.

Here's what we usually do:

Option 1: Screenshot → LLM

Drop an image into Claude or ChatGPT asking it to "recreate this." The result? The LLM guesses styles. Is that padding 16px or 24px? Is that blue #3b82f6 or #2563eb? Is the shadow 0 2px 8px or 0 4px 12px? Pure hallucination.

Option 2: DevTools → Copy Styles

Open inspector, right-click, "Copy styles." You get 200 properties including -webkit-tap-highlight-color and orphans: 2. Good luck extracting what actually matters.

Option 3: Figma Inspect

Great if you have the original Figma file. But Figma shows designer intent, not browser reality. That auto layout with 16px gap might render completely differently in CSS.

None of these give you what you actually need: the exact computed styles as the browser sees them.

The Design Loop

Here's a workflow that actually works:

┌─────────────────────────────────────────────────┐
│                                                 │
│   Reference UI ──► Snapshot ──► LLM ──► Code    │
│        ▲                                  │     │
│        │                                  ▼     │
│        └─────── Review in Browser ◄───────┘     │
│                                                 │
│                    REPEAT                       │
└─────────────────────────────────────────────────┘

Iteration 1: Capture the reference → LLM generates first version

Iteration 2: Capture your result + reference → "find the differences"

Iteration N: Refine until pixel-perfect

The key insight: instead of screenshots, we capture structured snapshots of computed styles. Tools like E2LLM can extract the actual CSS the browser computed — not what the developer wrote, not what Figma intended, but what pixels actually rendered. I covered the underlying schema in Runtime Snapshots #7 — Inside SiFR: The Schema That Makes LLMs See Web UIs.

Why Structured Snapshots Beat Screenshots

Screenshot	Structured Snapshot
LLM guesses: padding 16px?	Exact: `padding: 24px 32px`
"Looks kinda blue"	`#3b82f6` with `opacity: 0.9`
Can't see hover states	`actionType: "clickable"`
No hierarchy info	Full DOM tree with nesting
Font looks bold-ish	`font-weight: 600`, `font-family: Inter`

When you give an LLM structured data instead of pixels, it stops guessing and starts implementing. This is the same principle behind why static code analysis falls short — runtime state matters.

Practical Example: Cloning a Product Card

Scenario: Client points at an Amazon product card — "I want exactly this."

Step 1: Capture the reference

Using E2LLM browser extension, click on the element to get:

{
  "tag": "div",
  "computedStyles": {
    "padding": "16px",
    "borderRadius": "8px",
    "boxShadow": "0 2px 8px rgba(0,0,0,0.1)",
    "backgroundColor": "#ffffff"
  },
  "children": [
    {
      "tag": "img",
      "computedStyles": {
        "width": "100%",
        "aspectRatio": "1",
        "objectFit": "contain"
      }
    },
    {
      "tag": "h3",
      "text": "Product Title",
      "computedStyles": {
        "fontSize": "14px",
        "fontWeight": "400",
        "lineHeight": "1.4",
        "color": "#0f1111"
      }
    }
  ]
}

Step 2: Prompt the LLM

"Create a React component ProductCard with these exact styles. Use Tailwind where classes match exactly, inline styles for precise values that don't map to Tailwind."

Step 3: LLM generates code

function ProductCard({ image, title, price }) {
  return (
    <div
      className="p-4 rounded-lg bg-white"
      style={{ boxShadow: '0 2px 8px rgba(0,0,0,0.1)' }}
    >
      <img
        src={image}
        className="w-full aspect-square object-contain"
      />
      <h3
        className="text-sm font-normal leading-snug"
        style={{ color: '#0f1111' }}
      >
        {title}
      </h3>
    </div>
  );
}

Step 4: Capture your result

Open your component in the browser, capture it the same way.

Step 5: Diff prompt

"Compare these two snapshots. First is the reference, second is my implementation. List style differences and suggest fixes."

LLM responds:

Differences found:
- lineHeight: reference 1.4 vs yours 1.375 (leading-snug)
- Consider using style={{ lineHeight: '1.4' }} for exact match

Repeat until the diff is empty.

The Diff Mode: Two Pages Side by Side

For complex pages, you can capture both simultaneously:

Reference Site          Your Implementation
      │                         │
      ▼                         ▼
  [Capture]                 [Capture]
      │                         │
      └────────► LLM ◄──────────┘
                  │
                  ▼
          "Differences:
           - padding: 24px vs 16px
           - font-weight: 600 vs 400
           - border-radius: 12px vs 8px"

This turns subjective "looks close enough" into objective "here are the 3 remaining differences." Similar to how we use snapshots for semantic regression detection, but for design fidelity instead of functionality.

Why This Loop Works

Measurable progress — each iteration shrinks the diff
Objectivity — no more "it looks similar," just concrete numbers
Documentation — snapshots save as artifacts for future reference
In-context learning — the LLM learns from its mistakes within the session

This is the same loop pattern I described in Runtime Snapshots #10 — The Loop, applied to design workflow instead of QA.

When This Is Especially Useful

Agency work: Client shows competitor's site as reference
Redesigns: Updating UI while preserving the "feel"
Design systems: Extracting tokens from existing UI
Responsive work: Capture at different viewports → LLM adapts breakpoints

Try It Yourself

Install E2LLM for Chrome (also available for Firefox)
Open any site you want to recreate
Click the extension icon → Select Element → click a component
Paste the JSON into Claude or ChatGPT: "Recreate this as a React component"
Render your result, capture it
Send both snapshots: "Find the differences"
Fix and repeat

The gap between "reference" and "implementation" becomes a number that shrinks to zero. That's the Design Loop.

This is part of the Runtime Snapshots series, exploring how structured browser data changes the way we work with LLMs for frontend development. Check out E2LLM to try it yourself.

🧩 Runtime Snapshots #10 — The Loop: E2LLM in Production

Alechko — Fri, 09 Jan 2026 08:53:19 +0000

Your CI is green. Your users can't checkout.

This is the story of every team that trusted tests more than reality.

The Problem With "Ship and Pray"

Traditional deployment flow:

Code → Tests → Deploy → Hope → (Bug report 3 days later)

Tests check if code works. They don't check if users can use it.

A promo banner covering the checkout button? Tests pass.

Mobile layout pushing CTA below the fold? Tests pass.

Third-party widget blocking the form? Tests pass.

All green. Ship it. Break production.

The Loop

We built a different flow:

SHIP  -->  CAPTURE  -->  INTERPRET  -->  FIX  -->  REPEAT
  |           |              |            |           |
  v           v              v            v           v
Code hits   E2LLM        LLM analyzes  Actionable  Every deploy,
production  snapshots    with context  specific    every page
            real UI                    guidance

No screenshots. No guessing. Real state → Real answers.

Step 1: Ship

You merge. CI runs. Tests pass. Deploy triggers.

Nothing new here. The difference is what happens next.

Step 2: Capture

Post-deploy, E2LLM captures what users actually see.

Not the source code.

Not the accessibility tree.

Not a screenshot.

The rendered DOM—with salience scoring.

{
  "page": "/checkout",
  "viewport": "375x667",
  "elements": [
    {
      "selector": "#pay-button",
      "salience": 95,
      "visible": true,
      "occluded_by": "#promo-banner",
      "occlusion_percent": 60
    },
    {
      "selector": "#promo-banner",
      "salience": 70,
      "position": "fixed",
      "z_index": 999
    }
  ]
}

This is SiFR—Structured Interface Representation.

~400KB instead of megabytes. Zero schema overhead.

Step 3: Interpret

Feed the snapshot to any LLM. No special integration needed.

Prompt:

Analyze this checkout page snapshot.
Identify any UX blockers for completing purchase.

Response:

CRITICAL: Pay button (#pay-button) is 60% occluded by
promo banner (#promo-banner) on mobile viewport.

Users cannot reliably tap the primary CTA.

Suggested fix: Add margin-bottom to promo banner
or reduce z-index below checkout form.

The model doesn't guess. It sees what the user sees.

Step 4: Fix

You get:

What is broken (button occluded)
Why it's broken (z-index, fixed positioning)
How to fix it (specific CSS change)
Where it happens (mobile viewport only)

Not "button might be hard to click."

Not "consider checking mobile layout."

Actionable. Specific. Reproducible.

Step 5: Repeat

This isn't a one-time debug session. It's a loop:

// Post-deploy hook
async function checkDeployment(pages) {
  for (const page of pages) {
    const snapshot = await e2llm.capture(page);
    const analysis = await llm.analyze(snapshot);

    if (analysis.hasBlockers) {
      await alert.send({
        severity: 'critical',
        page: page,
        issues: analysis.blockers
      });
    }
  }
}

Every deploy. Every critical page. Automated.

What This Catches

                          E2E Tests    Visual Diff    The Loop
                          ---------    -----------    --------
Button covered by banner     Pass         Maybe        ALERT
CTA pushed off viewport      Pass         Pass         ALERT
Mobile-only layout break     Miss         Noise        ALERT
Third-party widget overlap   Pass         Pass         ALERT
Font size change            Ignore       500 alerts    Ignore

The difference: we check meaning, not pixels.

Integration Points

The Loop fits into existing infrastructure:

CI/CD:

# .gitlab-ci.yml
post_deploy_check:
  stage: verify
  script:
    - e2llm capture --pages /checkout,/signup,/dashboard
    - e2llm analyze --alert-on critical

Playwright:

await page.goto('/checkout');
const snapshot = await page.evaluate(() => 
  window.e2llm.capture({ selector: 'body' })
);

Monitoring:

// Cron: every hour
const pages = ['/checkout', '/signup', '/pricing'];
await Promise.all(pages.map(p => e2llm.checkAndAlert(p)));

The Mindset Shift

Old: "Tests pass, we're good."

New: "Tests pass, now let's check reality."

Tests verify intent.

The Loop verifies experience.

Both matter. But only one catches the banner blocking your checkout.

TL;DR

Ship → Capture → Interpret → Fix → Repeat

Capture real UI state with E2LLM (salience-scored, token-efficient)
Interpret with any LLM (Claude, GPT, Llama—your choice)
Fix with actionable, specific guidance
Repeat on every deploy, automatically

Your tests check if code works.

The Loop checks if users can use it.

👉 e2llm.dev — Free, open source, installs in seconds.

📖 Previous in series: Runtime Snapshots #9 — SiFR Deep Dive

Tags: #e2llm #qa #testing #devops #frontend #cicd #webdev