DEV Community

Cover image for I built an MCP server that lets AI agents use my Windows apps while I keep typing
imper
imper

Posted on

I built an MCP server that lets AI agents use my Windows apps while I keep typing

Most "computer use" tools take over the machine. The pointer moves, windows jump to the front, and you wait until the agent is done. I wanted an agent to work in one Windows app while I kept typing in another, so I built Sidehand.

What it does

Sidehand is an MCP server (plus a CLI and a pi extension) for Windows 10 and 11. Claude, Codex, Cursor or any MCP client can read and operate Windows apps that sit behind the window you are using. It types, clicks, picks from lists and reads the screen as text. Your mouse does not move and your keyboard stays with you.

What the agent sees

$ sidehand tree "Contoso Orders"
window: Contoso Orders  (18 elements)
  e4 Edit "Customer"
  e12 Button "Submit order"
$ sidehand act e4 type "Blue-B Studio"
$ sidehand act e12 click
Enter fullscreen mode Exit fullscreen mode

One short line per control. Reading a window takes about 4 to 5 times fewer tokens than agent-desktop on typical apps, and about 20 times fewer on an Edge page.

Benchmark

The method is in docs/benchmark.md. In the benchmark task:

  • The note I was typing in kept the keyboard the whole time: 0 s lost, no window came forward, the cursor never moved.
  • The task took 277 tokens, against 2,488 to 13,425 for the other tools.
  • It finished in 2.0 s. Windows-MCP and Terminator, the other tools that finished it, took 65.5 s and 68.0 s.

When an app pulls itself forward

Some apps bring their own window to the front while they handle a click. Windows gives a background process no way to forbid that, so Sidehand watches the foreground while an action runs and puts your window back as it happens. In my tests on the real desktop that took 11 to 13 ms. A key you press in that moment can still be lost, and the result tells the agent when it happened.

Try it

claude mcp add sidehand -- npx -y sidehand mcp
Enter fullscreen mode Exit fullscreen mode

Cursor, Codex and Claude Desktop config are in the README. For the CLI: npm install -g sidehand.

Limits

  • WPF: right-click, hover and drag do not work in the background.
  • Keys: typing into a text field of a browser window works in the background (tested in Edge). Other keys and shortcuts in WPF and browser windows need --allow-foreground, which is your permission to take the foreground for the keys. Enter on a button or link works without it.
  • Chrome and Edge: an HTML <select> drop-down cannot be operated in the background.
  • Games that read raw mouse input ignore posted clicks. Capture and OCR still work.
  • Apps running as administrator need an agent running as administrator.
  • Windows only.

License

Sidehand is source-available under PolyForm Shield 1.0.0: free for any use, including at work, except building or offering a product that competes with Sidehand. It is not OSI open source.

Repo: https://github.com/Blue-B/sidehand

Bug reports for apps that do not behave in the background are very welcome.

Top comments (0)