DEV Community: Safiyev Marat

How I Built an AI Agent That Controls Browsers, Terminals, and Desktop Apps

Safiyev Marat — Sat, 19 Sep 2026 10:23:23 +0000

How I Built an AI Agent That Controls Browsers, Terminals, and Desktop Apps

An LLM can tell you what to do.

But what happens when you ask it to actually do something on your computer?

Open a browser. Create a document. Execute a terminal command. Manage files. Verify the result.

A language model alone cannot reliably handle all of this.

That's why I built HeyAgent, an open-source AI agent designed to interact with real desktop environments.

Here's how its architecture works.

1. The Core Architecture

HeyAgent isn't just an LLM connected to a few tools.

It uses a structured execution pipeline:

User Request
     |
     v
Context & Memory
     |
     v
Router
     |
     v
Planner
     |
     v
Mission Queue
     |
     v
Execution Engine
     |
     v
Tools
     |
     v
Verification
     |
     +---- Success ---> Response
     |
     +---- Retryable Failure ---> Execution Engine

Each component has a specific responsibility.

The model helps decide what needs to happen. The execution system handles how it happens.

2. Planning Before Execution

Imagine giving HeyAgent this request:

Open the browser, research a topic, and save the results to a file.

This isn't a single action.

It requires several steps:

Understand the request.
Determine which tools are needed.
Organize the actions.
Execute the workflow.
Verify the result.

HeyAgent uses routing and planning to prepare tasks before execution.

A mission queue coordinates the work, while the execution engine runs the required actions.

Depending on the task, execution can use a deterministic harness or an LLM-driven tool loop.

3. Connecting the LLM to a Real Computer

The execution engine connects the model to tools that interact with the operating system.

HeyAgent supports:

Browser: Navigate websites and interact with pages.
Desktop: Control the mouse, keyboard, windows, and applications.
Terminal: Execute shell commands.
Files: Read, create, and manage local files.
Integrations: Work with connected services, including Google Workspace.

This allows a single workflow to move between different environments.

For example, an agent can collect information through a browser, process it using a terminal command, and save the output to a local file.

These operations are coordinated through the agent runtime rather than being completely independent scripts.

4. Execution Isn't Completion

One important part of the architecture is verification.

A successful tool call doesn't necessarily mean the user's task is complete.

A browser action might succeed while the website returns an error.

A command might execute without producing the expected output.

HeyAgent includes a separate verification stage that evaluates execution evidence before the workflow reports completion.

If verification detects a retryable failure, execution can continue.

Execute
   |
   v
Verify
   |
   +-- Evidence passes --> Complete
   |
   +-- Retryable failure --> Continue execution

This doesn't guarantee that every task will succeed.

It creates an explicit distinction between performing an action and achieving the requested result.

I explored this problem in more detail in my previous article about false task completion.

5. Computer Control Needs Permissions

Giving an AI agent access to a real computer introduces obvious risks.

A tool capable of managing files or running commands can also perform destructive actions.

HeyAgent includes a permission system with configurable policies, including approval requirements and capability allowlists.

Sensitive operations can require explicit user confirmation before execution.

The goal is to make computer automation useful without treating every model-generated action as automatically authorized.

6. No Dependency on a Single LLM

Another design decision was to keep the architecture model-agnostic.

HeyAgent supports different model providers, including cloud services and local models through Ollama.

The system also includes model failover capabilities.

The reasoning model can change without requiring an entirely different computer-control architecture.

This separation makes it easier to experiment with different models while keeping the surrounding execution system consistent.

The Bigger Picture

Building an autonomous AI agent isn't just about writing better prompts.

It's about connecting several systems:

Planning, execution, computer control, permissions, state management, and verification.

HeyAgent brings these components together in a local-first architecture accessible through a CLI, desktop application, and Telegram bot.

It's still an evolving open-source project, and there's plenty of work ahead.

If you're interested in computer-use agents, automation, or agent architecture, feel free to explore the code and contribute.

GitHub: https://github.com/SAFIYEV/HeyAgent

⭐ If you find the project interesting, consider giving it a star.

I'd also love to hear how you're approaching execution and verification in your own AI agents.

Why AI Agents Say “Done” When the Task Actually Failed

Safiyev Marat — Tue, 11 Aug 2026 07:51:54 +0000

AI agents have a surprisingly simple reliability problem:

They often confuse performing an action with completing a task.

An agent clicks a button.

The tool call succeeds.

The model assumes the job is finished.

And then you get:

Done ✅

But maybe the page returned an error.

Maybe the file wasn't created.

Maybe the message wasn't sent.

Maybe the application never opened.

The action succeeded.

The task didn't.

The False Completion Problem

Imagine asking an agent:

“Create a Google Doc called Project Plan.”

A naive agent might:

Open Google Docs
Click “New document”
Enter the title
Report success

But clicking buttons isn't proof that the document actually exists.

For autonomous agents, this becomes a serious problem.

The longer the workflow, the worse it gets.

One incorrect assumption early in the process can make every step after it useless.

How HeyAgent Handles It

While building HeyAgent, I decided that executing the last action should not automatically mean the task is complete.

The basic workflow looks like this:


text
Request
   ↓
Plan
   ↓
Execute
   ↓
Verify
   ↓
Success

And if verification fails:

Execute
   ↓
Verify
   ↓
Failure detected
   ↓
Retry / continue execution

HeyAgent has a separate verification stage after execution.

Instead of trusting the model's assumption that something worked, the system checks for evidence that the expected result was actually produced.

Only after that verification passes should the task be considered completed.

Why This Matters

There is a big difference between:

tool_call = success

and:

user_goal = achieved

The first tells us that an operation ran.

The second tells us that the agent actually did what the user asked.

For simple chat assistants, this distinction may not matter much.

For agents controlling browsers, files, terminals, desktop applications, and external services, it matters a lot.

If we want AI agents to perform real work reliably, they can't just be good at taking actions.

They also need to understand whether those actions actually worked.

That's one of the reliability problems I'm trying to solve with HeyAgent.

HeyAgent is fully open source, so you can explore the architecture, test it, break it, and contribute.

⭐ GitHub: https://github.com/SAFIYEV/HeyAgent

I Built an Open-Source AI Agent That Actually Controls Your Computer

Safiyev Marat — Mon, 03 Aug 2026 15:27:18 +0000

AI agents are everywhere in 2026.

Most of them can answer questions, generate code, or automate simple workflows. But once you ask them to interact with a real computer—browsers, desktop applications, terminals, files, and external services—things quickly become unreliable.

That was the motivation behind HeyAgent.

The Problem

Most autonomous agents fail for one of three reasons:

They declare success before the task is actually finished.
They lose context during long, multi-step workflows.
They aren't designed to work with a real desktop environment.

I wanted to build an agent that behaves more like a real assistant instead of just another LLM wrapper.

What HeyAgent Does

HeyAgent is an open-source autonomous AI agent for computer control and workflow automation.

It can:

🖥️ Control desktop applications
🌐 Work inside browsers
📂 Read and manage files
💻 Execute terminal commands
🔗 Connect with external services
📱 Be controlled through CLI, Desktop UI, or Telegram
🧠 Plan and execute multi-step workflows
✅ Verify results before marking tasks as completed

Instead of blindly executing prompts, the agent plans, executes, validates the outcome, and only then reports success.

Reducing False Task Completion

One of the biggest problems I noticed in existing AI agents is false task completion.

Many agents click a button, assume everything worked, and immediately report success.

In reality, something may have failed several steps earlier.

HeyAgent performs additional verification after critical actions to reduce false positives and improve reliability during long-running workflows.

Built with AWS Support

HeyAgent has been significantly accelerated thanks to the support of AWS.

AWS has provided the project with cloud infrastructure, GPU computing resources, and access to modern AI services that made rapid experimentation possible throughout development.

From running GPU workloads to evaluating different LLMs and AI models, AWS has been an important part of the engineering process. Their continued support allows us to iterate faster, test new ideas, and focus on improving the product instead of worrying about infrastructure limitations.

We're genuinely grateful to AWS for supporting open-source innovation and continuing to help us build HeyAgent.

Tech Stack

TypeScript
Node.js
Model-agnostic architecture
Cloud and local LLM support
Plugin-based tool system
Multi-step execution engine

The goal isn't to lock developers into one model.

Whether you prefer cloud providers or local models, HeyAgent is designed to work with both.

Why Open Source?

AI agents shouldn't become black boxes.

Developers should be able to inspect the reasoning flow, build their own tools, contribute improvements, and adapt the platform to their own workflows.

That's why HeyAgent is fully open source.

What's Next?

Development is moving fast, and we're actively working on:

Better planning and reasoning
More reliable desktop automation
A richer plugin ecosystem
Improved routing and execution
Support for additional AI models and integrations

We'd Love Your Feedback

If you're building AI agents, automation tools, or desktop software, I'd love to hear your thoughts.

Feedback, ideas, bug reports, and pull requests are always welcome.

⭐ GitHub: https://github.com/SAFIYEV/HeyAgent

A huge thank you again to AWS for supporting HeyAgent with cloud infrastructure, compute resources, and AI services. Their continued support has played an important role in bringing this project to life.