I wanted to solve a very specific problem: AI chats can reason, but they usually cannot actually operate the Windows computers where the work lives.
So I built YourHand - a Windows control layer that connects supported AI clients to one or more Windows PCs.
The goal is simple:
One AI chat should be able to work across the computers you already use.
That means opening applications, reading the screen, clicking, typing, managing files, running commands, and coordinating tasks across multiple machines - without forcing everything into a browser-only workflow.
Why I built it
My daily work is spread across several Windows machines. Different computers have different applications, files, sessions, and workloads. A normal AI assistant can tell me what to do, but I still have to jump between machines and execute every step manually.
I wanted the interaction to feel closer to this:
Me: Open Revit on PC-075 and check whether the project is already running.
AI: [checks the device, opens the app if needed, reports the result]
Me: Now open Chrome on BeDo-Tower and test the latest build.
AI: [works on the second machine without losing the first context]
The important part is not the mouse click itself. The important part is persistent, account-scoped access to multiple real Windows computers from the same conversation.
The architecture
The system is split into a few layers:
AI client
YourHand MCP / plugin
Control plane
Capability router
Windows agent
OS / browser / UI / files / commands
The control plane handles account identity, device ownership, sharing, routing, and request coordination.
Each Windows machine runs a lightweight agent. The agent exposes capabilities such as:
- window and application control
- keyboard and mouse input
- screen observation
- file access
- command execution
- browser control
- device health and telemetry
The agent is designed to run quietly in the background. The user should not need to keep a terminal window open just to make the computer controllable.
Why a capability router matters
One of the biggest lessons from building this is that desktop automation should not depend on a single technique.
A click can be executed through several possible paths:
- direct OS or application APIs
- browser CDP when the target is a web page
- semantic Windows UI Automation
- pixel-based interaction
- foreground input as a fallback
- vision-assisted fallback when the UI cannot be addressed semantically
So instead of treating "click this button" as a single primitive, YourHand routes the task to the most appropriate capability.
That makes the system more robust than relying entirely on screenshots and blind cursor movement.
Multi-device was a first-class requirement
A lot of automation tools assume one session, one machine, one active task.
That was not enough for what I wanted.
YourHand treats devices as explicit resources tied to the user's account. A conversation can target different computers by name, and the control plane routes each request to the correct agent.
That also made sharing important.
The design goal is that one Windows agent can remain installed while access is granted to another authorized account. The device does not need a second competing agent just because another team member needs access.
Authentication without handing over passwords
I did not want users copying AI-provider passwords or long-lived manual tokens into the desktop agent.
The current flow uses Google sign-in and OAuth-based authorization for the YourHand account. Devices are then paired to that account and can be explicitly shared.
The separation is important:
- the AI client gets authorization to call YourHand
- YourHand decides which devices that account may use
- the Windows agent only accepts work routed through the control plane
Observability matters more than I expected
Desktop automation fails in messy ways.
A command can time out even though the action happened. A window can disappear between discovery and input. A browser can be alive while the target tab is not. A device can reconnect with a stale socket.
Because of that, I added telemetry around calls: success/failure, execution method, latency, retries, and device health.
This has been one of the most valuable parts of the project. Without observability, every failure looks like "the AI clicked the wrong thing." With telemetry, you can distinguish routing failures, UI failures, transport failures, and application failures.
Open source, but not just a demo
I published a community repository because I want the Windows/AI-control layer to be inspectable and extensible.
The project is still in beta, but the target is not a toy demo. The architecture is being built around:
- multiple Windows devices
- multiple authorized users
- account isolation
- browser and native UI control
- background execution
- recovery after restarts
- device telemetry
- controlled sharing
The hard part is not making a mouse move once. The hard part is making the whole path reliable enough that a user can trust the system across real machines.
What I am testing now
The current beta is focused on three things:
- Reliability - fewer fragile UI assumptions and better recovery.
- Speed - choosing direct APIs or semantic automation before falling back to screenshots.
- Multi-device workflows - keeping one conversation useful across several computers.
I am also testing the onboarding flow for people who have never configured an MCP server or a Windows automation agent before.
Try it
YourHand is currently free during beta.
Website: https://yourhand.wolvexai.com/
Open-source community repository: https://github.com/archahmedzaki/YourHand-Community
I would especially like feedback from people who build developer tools, IT automation, MCP integrations, or Windows desktop software.
The question I am trying to answer is:
What would make you trust an AI chat enough to let it work across your real computers?
Top comments (0)