DEV Community: capman

Beyond Browser Automation: How Teams Are Actually Solving Agent Reliability

capman — Thu, 11 Jun 2026 19:19:39 +0000

AI agents are becoming increasingly capable, yet many production failures have nothing to do with intelligence. A button moves, a modal appears, a page loads differently, and an automation that worked yesterday suddenly breaks.

This raises an important question: is browser automation the right abstraction for reliable AI systems?

As teams move beyond demos and into production, many are exploring alternatives such as APIs, tool ecosystems, workflow engines, and hybrid architectures. What's emerging isn't a single winning approach, but a broader shift toward building AI systems that prioritize reliability over novelty.

Stage 1: Browser Automation

For many teams, browser automation is the natural starting point.
The philosophy is simple: If humans can perform a task through a UI, AI should be able to do the same.

Frameworks like:

make this surprisingly easy.

This is also the philosophy behind many recent "computer-use" agents. The appeal is obvious.

No APIs.
No integration work.
Just use the application exactly like a human would.

But over time, many teams discover that they're not trying to automate browsers. They're trying to automate systems. And the browser simply happens to sit in the middle.

As reliability becomes important, that extra layer starts to feel increasingly expensive.

Stage 2: Moving Closer to the Source

Many teams eventually realize that communicating directly with systems is fundamentally simpler than navigating interfaces.

Instead of asking:

"Where is the button?"

they start asking:

"Which operation should be executed?"

This is the philosophy behind platforms like:

The shift may seem small, but it removes entire categories of failures.

No screenshots.
No waiting for pages.
No brittle selectors.
No UI redesign problems.

Stage 3: Tool Ecosystems

As integrations multiply, another problem begins to emerge.
Tool sprawl.

One integration becomes ten.
Ten become hundreds.
Eventually, selecting the right capability becomes harder than executing it.

This is partly why protocols such as:

have attracted so much attention recently.

They focus on making tools easier to expose and consume. Ironically, solving interface navigation introduces a new kind of navigation problem:

Capability navigation.

Stage 4: The Return of Workflows

Another interesting trend is the resurgence of workflow engines.

Tools like:

embrace a different philosophy.

Instead of giving agents unlimited freedom, they constrain them inside predictable flows. This may sound less exciting. But production systems often value predictability more than autonomy. Users don't care how creative your architecture is.

They care whether it works.

Stage 5: Hybrid Architectures

Perhaps the most interesting trend is that many teams seem to be combining everything.

Increasingly, reliability seems to come from layers rather than a single technology.
For example:

Use APIs whenever possible.
Fall back to browser automation when necessary.
Escalate uncertain situations to humans.

This philosophy is visible across many modern agent frameworks, including:

What's interesting is that browser automation doesn't disappear.

It simply becomes a fallback rather than the default. And perhaps that's the biggest lesson I'm seeing across the ecosystem. Reliable systems aren't replacing one approach with another. They're layering multiple approaches together

GitHub: github.com/Hobbydefiningdoctory/capman
Capman-site: capman

capman v0.6.2 — TypeScript, MIT licence, dual CJS/ESM, zero runtime dependencies beyond zod.

Why Teaching AI to Click Buttons Is a Broken Abstraction

capman — Tue, 09 Jun 2026 23:02:57 +0000

We've all experienced it.

You automate a repetitive workflow, only for everything to break because someone moved a button or redesigned a page. Surprisingly, many AI agents suffer from the exact same problem.

Instead of understanding the underlying task, they're often forced to interact with software the same way humans do: by clicking buttons, reading screens, and navigating menus.

This approach works.
But it raises an important question:

Are we teaching AI systems to solve problems, or are we simply teaching them to use software designed for humans?

By the end of this article, we'll explore why UI navigation has become one of the biggest bottlenecks in AI agents and why many developers believe the industry is solving the wrong problem.

A Simple Task Isn't Really Simple

Imagine someone wants to check the status of an order.

For a browser-based AI agent, the process might look something like this:

What seems like a simple request actually requires multiple steps. And every step introduces opportunities for failure.

The UI Was Built For Humans

Graphical user interfaces exist because humans need visual representations of information. Buttons, menus, and forms are abstractions created for us.

Computers themselves don't need buttons.

Yet many modern AI agents spend enormous amounts of computation trying to interpret screenshots, locate interface elements, and simulate mouse clicks.
They're essentially pretending to be humans.

We've Seen This Problem Before

In Robotic Process Automation (RPA). Large enterprises invested heavily in automation workflows that depended on screen layouts and specific button locations.

Then a website redesign happened.
Or a menu moved.
Or a label changed.

And suddenly entire workflows stopped working.

Fragile automation became one of the biggest challenges facing RPA systems.
Many AI agents are repeating the same pattern—only with language models replacing traditional scripts.

Browser Agents Pay a Heavy Price

Research from companies building browser agents has also highlighted challenges such as:

Slow execution
Increased token costs
Fragility
Ambiguous actions

The Industry Is Optimizing The Wrong Layer

Today's AI systems are incredibly powerful.

Yet some of them spend most of their time performing tasks like:

Finding buttons.
Waiting for pages to load.
Reading screenshots.
Navigating menus.

These aren't the user's actual goals.
Users don't want a button clicked.
Users want a problem solved.
There's a difference.

The Irony

Humans navigate interfaces because we have no alternative.

Machines do.

And yet we're investing enormous amounts of compute teaching AI systems how to behave like users instead of asking whether user interfaces are even the right abstraction for machines.

The result is an interesting paradox:

Some of the world's most advanced AI models spend their time doing something web browsers have been doing since the 1990s.

GitHub: github.com/Hobbydefiningdoctory/capman
Capman-site: capman

capman v0.6.2 — TypeScript, MIT licence, dual CJS/ESM, zero runtime dependencies beyond zod.

Capman: Stop Routing Every Intent Through an LLM

capman — Sun, 12 Apr 2026 10:31:26 +0000

Most AI apps send every user message to a language model — even when the intent is completely predictable. capman intercepts the 80% you already know how to answer, resolves them in under 2 ms at zero token cost, and lets the LLM handle only the queries that genuinely need it.

The Problem

When a user types "what time is it?" into your AI app, this is what happens:

User input
  → serialize message
  → HTTP request to LLM API     ← 800–1500 ms
  → token cost (~$0.001–$0.01)
  → parse function call JSON
  → invoke time handler

You paid an LLM to decode "what time is it" into a clock call. It knew the answer in 1 ms. You waited 1,200 ms anyway.

Standard function calling routes every query through the model — by design. For ambiguous, open-ended input, that's the right call. For well-defined intents like "check availability", "go to settings", or "get my orders", it's pure overhead.

What is capman?

capman is a TypeScript library that sits in front of your LLM. It reads a capability manifest — a machine-readable list of everything your app can do — and matches user queries against it using weighted keyword scoring.

If it finds a confident match: it resolves the query directly. No LLM. No network. No tokens.
If it doesn't: it escalates to your LLM automatically.

import { CapmanEngine, readManifest } from 'capman'

const engine = new CapmanEngine({
  manifest: readManifest(),
  mode: 'balanced',          // keyword-first, LLM only when needed
  llm: async (prompt) => callYourLLM(prompt),
})

const result = await engine.ask('Check availability for blue jacket')

console.log(result.match.capability?.id)  // 'check_product_availability'
console.log(result.resolvedVia)           // 'keyword' | 'llm' | 'cache'
console.log(result.trace.totalMs)         // 1.6

How It Works

Every engine.ask() call passes through six steps:

1. Cache check — normalized query key lookup. Hit → return in <1 ms.

2. Keyword scorer — every capability in your manifest is scored 0–100 across three weighted sources:

Source	Weight	Signal
Example sentences	60 pts	Strongest — direct intent overlap
Capability description	30 pts	Secondary — synonym coverage
Capability name	10 pts	Weakest — tiebreaker only

3. Branch decision — confidence ≥ 50%? Direct resolve. Below threshold? LLM escalation (rate-limited + circuit-breaker protected).

4. Privacy enforcement — public, user_owned, or admin checked before any API call fires.

5. Resolver — api (fetch + retry + timeout), nav (URL template), or hybrid (both in parallel).

6. Learning + trace — full ExecutionTrace returned with step-by-step timing. Pre-boost result recorded to prevent feedback loops.

Three Modes, One Decision

cheap      → keyword only, <2 ms, $0 — for high-volume known intents
balanced   → keyword first, LLM if <50% confidence — default
accurate   → LLM first, keyword fallback — for ambiguous open input

The mode is set per-engine instance. Switch by swapping the constructor option.

capman vs Standard Function Calling

The fundamental difference is when the LLM is involved:

	LLM Function Calling	capman
Latency	800–1,500 ms	<2 ms (direct)
Token cost	~$0.001–$0.01/call	$0 on direct match
Failure rate	~25% (hallucinations, timeouts)	0% on direct match
Privacy enforcement	Manual	Built-in per capability
Caching	External setup	Memory / File / Combo built-in
Tracing	Custom logging needed	Automatic on every call

capman doesn't replace your LLM — it protects it. The model gets fewer calls, better-framed prompts, and none of the deterministic work it was doing before.

Demo: The Weather Query

The demo runs a battle test: the same query submitted three times in a row, both engines running in parallel. Here's what came back.

The Query

"what is the weather"

Results

Round 1 — cold start:

LLM: 1,151.17 ms ✓ Success → correct tool
capman: 1.61 ms — matched "weather" exactly — 715× faster

Round 2 — learning kicks in (+1 pt boost):

LLM: 851.51 ms
capman: 0.59 ms — boost applied from prior hit — 1,443× faster

Round 3 — boost grows (+2 pts):

LLM: 931.02 ms
capman: 0.78 ms — 1,193× faster

Why Does the Latency Drop on Rounds 2 and 3?

The learning store tracks keyword → capability → hitCount. On each successful match, it builds a boost index. On the next call:

boost = min(15, log₂(hitCount + 1) × 2)

More hits → logarithmically larger boost → capability scores higher faster → match resolves sooner in the scorer loop. The cap is +15 points total. The pre-boost result is what gets recorded (not the post-boost result) — this prevents a feedback loop where boosted winners accumulate hits and permanently displace keyword matches.

A Capability Definition

Everything capman does flows from a structured manifest. Here's a full capability:

// capman.config.js
module.exports = {
  app: 'my-store',
  baseUrl: 'https://api.my-store.com',
  capabilities: [
    {
      id: 'check_product_availability',
      name: 'Check product availability',
      description: 'Check stock and pricing for a product by name or ID.',
      examples: [
        'Is the blue jacket in stock?',
        'Check availability for product 42',
        'Do you have size M?',
      ],
      params: [
        {
          name: 'product',
          description: 'Product name or ID',
          required: true,
          source: 'user_query',   // extracted from the query
        },
      ],
      returns: ['stock', 'price', 'variants'],
      resolver: {
        type: 'api',
        endpoints: [{ method: 'GET', path: '/products/{product}/availability' }],
      },
      privacy: { level: 'public' },
    },
  ],
}

Then:

npx capman generate   # → manifest.json
npx capman validate   # → check all capabilities
npx capman demo       # → live test with sample queries

What capman Does Not Replace

capman handles the routing layer. It doesn't:

Generate creative responses
Summarize or reason over content
Handle open-ended questions with no predictable shape
Replace conversational context management

For any of those, the LLM still runs — capman just makes sure it runs only when it needs to.

Install

npm install capman

npx capman init       # create capman.config.js
npx capman generate   # generate manifest.json
npx capman run "your query" --debug

GitHub: github.com/Hobbydefiningdoctory/capman
Capman-site: capman

capman v0.4.5 — TypeScript, MIT licence, dual CJS/ESM, zero runtime dependencies beyond zod.

I built a tool that lets AI agents interact with your app without navigating the UI

capman — Mon, 30 Mar 2026 19:43:21 +0000

When I started thinking about how AI agents interact with applications, something bothered me.

Today, when an AI agent needs to answer "are there seats available for Friday?" — it navigates your app like a tourist with no map:

AI clicks → Home → Explore → Events → Category → Availability

That's slow, wasteful, and it exposes parts of your app the AI was never supposed to see.

So I built capman — a Capability Manifest Engine that gives AI agents a structured map of what your app can do, and shows you exactly why it made every decision.

The core idea

Your app publishes a capability manifest — a machine-readable list of everything it can do, what API to call, and what data scope is allowed.

// capman.config.js
module.exports = {
  app: 'my-app',
  baseUrl: 'https://api.my-app.com',
  capabilities: [
    {
      id: 'check_availability',
      name: 'Check availability',
      description: "'Check if a product or slot is available on a given date.',"
      examples: [
        'Are there seats available Friday?',
        'Check availability for blue jacket',
      ],
      params: [
        { name: 'item',    description: "'Item name or ID', required: true,  source: 'user_query' },"
        { name: 'date',    description: "'Date to check',   required: false, source: 'user_query' },"
      ],
      returns: ['available', 'count', 'price'],
      resolver: {
        type: 'api',
        endpoints: [{ method: 'GET', path: '/availability/{item}' }],
      },
      privacy: { level: 'public' },
    },
  ],
}

The AI reads this manifest and goes directly to the answer — no navigation, no guessing.

The part I'm most proud of: the execution trace

Every query returns a full execution trace. No black box.

const engine = new CapmanEngine({ manifest, baseUrl: 'https://api.my-app.com' })
const result = await engine.ask('Are there seats available Friday?')

console.log(result.trace)
// {
//   query: 'Are there seats available Friday?',
//   candidates: [
//     { capabilityId: 'check_availability', score: 100, matched: true  },
//     { capabilityId: 'get_orders',         score: 12,  matched: false },
//     { capabilityId: 'navigate_to_screen', score: 0,   matched: false },
//   ],
//   reasoning: [
//     'Matched "check_availability" with 100% confidence',
//     'Rejected: get_orders (12%)',
//     'Resolved via: keyword',
//     'Extracted params: item=seats, date=Friday',
//   ],
//   steps: [
//     { type: 'cache_check',   status: 'miss', durationMs: 0 },
//     { type: 'keyword_match', status: 'pass', durationMs: 1, detail: 'confidence: 100%' },
//     { type: 'privacy_check', status: 'pass', durationMs: 0, detail: 'level: public'    },
//     { type: 'resolve',       status: 'pass', durationMs: 2, detail: 'via api'          },
//   ],
//   totalMs: 4,
// }

This answers the question every AI developer eventually asks:

"Why did it pick this capability?"
"Why wrong parameters?"
"Why did it fail?"

Today with LangChain, OpenAI function calling, or custom agents — there's no clear answer. Logs are scattered, debugging is painful.

capman makes AI execution debuggable like backend code.

Three matching modes — control cost vs accuracy

// cheap — keyword only, free, fast
const engine = new CapmanEngine({ manifest, mode: 'cheap' })

// balanced — keyword first, LLM fallback if confidence < 50% (default)
const engine = new CapmanEngine({ manifest, mode: 'balanced', llm: myLLM })

// accurate — LLM first, always correct
const engine = new CapmanEngine({ manifest, mode: 'accurate', llm: myLLM })

The LLM function is plug-and-play — works with Anthropic, OpenAI, or any model:

const engine = new CapmanEngine({
  manifest,
  mode: 'balanced',
  llm: async (prompt) => {
    const res = await anthropic.messages.create({
      model: 'claude-sonnet-4-20250514',
      max_tokens: 500,
      messages: [{ role: 'user', content: prompt }],
    })
    return res.content[0].text
  },
})

Privacy enforcement — per capability, not per request

{
  id: 'get_my_orders',
  privacy: { level: 'user_owned' },  // blocked without auth
}

{
  id: 'get_admin_report',
  privacy: { level: 'admin' },       // blocked without admin role
}

Pass auth context once — capman enforces it before resolution:

const engine = new CapmanEngine({
  manifest,
  auth: {
    isAuthenticated: true,
    role: 'user',
    userId: 'user-123',  // auto-injected into session params
  },
})

See it live in 30 seconds

npx capman demo

That runs a live demo against a sample e-commerce app — no config needed. You'll see matched capabilities, execution traces, candidate scores, and API calls constructed in real time.

Or run a query against your own manifest:

npx capman init          # creates capman.config.js
npx capman generate      # generates manifest.json
npx capman run "are there seats available Friday?" --debug

Get started

npm install capman

GitHub: github.com/Hobbydefiningdoctory/capman
npm: npmjs.com/package/capman
Capman-site: capman

capman v0.4.5 — TypeScript, MIT licence, dual CJS/ESM, zero runtime dependencies beyond zod.