<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Keyaburi</title>
    <description>The latest articles on DEV Community by Keyaburi (@keyaburi).</description>
    <link>https://dev.to/keyaburi</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4252848%2F933c7783-babe-40f5-b5e1-62368d02cba4.png</url>
      <title>DEV Community: Keyaburi</title>
      <link>https://dev.to/keyaburi</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/keyaburi"/>
    <language>en</language>
    <item>
      <title>I wanted to game while my AI agent ran tests — so I built it a cursor of its own</title>
      <dc:creator>Keyaburi</dc:creator>
      <pubDate>Thu, 19 Nov 2026 04:58:11 +0000</pubDate>
      <link>https://dev.to/keyaburi/i-wanted-to-game-while-my-ai-agent-ran-tests-so-i-built-it-a-cursor-of-its-own-m22</link>
      <guid>https://dev.to/keyaburi/i-wanted-to-game-while-my-ai-agent-ran-tests-so-i-built-it-a-cursor-of-its-own-m22</guid>
      <description>&lt;h1&gt;
  
  
  I wanted to game while my AI agent ran tests — so I built it a cursor of its own
&lt;/h1&gt;

&lt;h2&gt;
  
  
  The problem
&lt;/h2&gt;

&lt;p&gt;I run coding agents on my home PC in the evening. They are genuinely useful: while I play a game, an agent can run a test suite, click through a desktop app, and report back.&lt;/p&gt;

&lt;p&gt;Except for one thing: &lt;strong&gt;the agent uses my mouse.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Screenshot-and-click computer use works by moving the one real system cursor and taking the foreground. That is fine when nobody is sitting at the PC. It is miserable when somebody is — mid-game, the cursor teleports, a window jumps to the front, and my input goes somewhere I never intended.&lt;/p&gt;

&lt;p&gt;I didn't want a second PC, and I didn't want a VM. I wanted one machine where a human and an agent work at the same time, in different windows, without fighting over a single pointer.&lt;/p&gt;

&lt;h2&gt;
  
  
  The obvious options, and why each one failed me
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. SendInput / coordinate clicking (what most computer-use agents do)&lt;/strong&gt;&lt;br&gt;
There is exactly one system cursor. Any design that injects "real" mouse input is, by construction, a design that steals it. No scheduling trick fixes that; it's the wrong layer.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. A VM or a second session&lt;/strong&gt;&lt;br&gt;
Isolation works, but it's heavy, and it changes the problem: the agent is no longer testing &lt;em&gt;my&lt;/em&gt; apps in &lt;em&gt;my&lt;/em&gt; session, with my files and my logged-in state. For some people that's the right answer. For "run tests on this PC while I use this PC," it wasn't mine.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. MouseMux&lt;/strong&gt;&lt;br&gt;
MouseMux is a real, well-made product: multiple independent cursors on one Windows desktop, and it now even ships an MCP "Input Mapper" so an AI agent can get its own virtual user. If several &lt;em&gt;humans&lt;/em&gt; share one PC — point of sale, trading desks, remote collaboration — stop reading and go buy it.&lt;/p&gt;

&lt;p&gt;It didn't fit my case, for three reasons:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;My "second user" is not a human with a second mouse. It's software. I don't need an input &lt;em&gt;device&lt;/em&gt; layer; I need an input &lt;em&gt;policy&lt;/em&gt; layer.&lt;/li&gt;
&lt;li&gt;I want to be able to audit everything the agent can do. A closed binary that emulates the Windows input layer is hard to reason about — and on a gaming PC, "emulates the input layer" is exactly the shape anti-cheat systems are suspicious of.&lt;/li&gt;
&lt;li&gt;I wanted a hard rule a product can't promise me: &lt;strong&gt;the agent never moves the real cursor. Ever. If it can't act without doing so, it refuses.&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;4. Virtual HID drivers (FakerInput and friends)&lt;/strong&gt;&lt;br&gt;
Same anti-cheat problem, worse: now there's a kernel-adjacent input driver on a machine that runs games with strict anti-cheat. I wasn't willing to install that on my gaming PC.&lt;/p&gt;

&lt;h2&gt;
  
  
  The insight
&lt;/h2&gt;

&lt;p&gt;Here's the thing I got wrong at first: I assumed an agent needs to &lt;em&gt;click&lt;/em&gt;, the way a human clicks.&lt;/p&gt;

&lt;p&gt;Most Windows applications don't actually require that. Native controls accept window messages (&lt;code&gt;BM_CLICK&lt;/code&gt;, &lt;code&gt;EM_*&lt;/code&gt; and friends). Modern frameworks expose UI Automation patterns — Invoke, Value, Selection and so on. Both work &lt;strong&gt;in the background, on a window that is not in the foreground, without moving the pointer at all.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;And for the controls that accept neither? The honest answer is not "fall back to the real mouse." The honest answer is &lt;strong&gt;"this one, I can't do."&lt;/strong&gt; A refusal is a feature: it keeps the guarantee intact, and it keeps the human's session untouched.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I built (overview)
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F21uls7l4aoa5zum9w6oy.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F21uls7l4aoa5zum9w6oy.png" alt=" " width="799" height="512"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A small, driverless, user-mode tool — one executable, no network calls — that gives an agent its own way to act:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Order of attempts, always the same:&lt;/strong&gt; window messages first, then UI Automation patterns, then an explicit refusal. There is no code path that switches to real-cursor input.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One approved window.&lt;/strong&gt; The agent can only act on a window a human has approved. Password fields and elevated (admin) windows are refused outright.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;An Arm switch.&lt;/strong&gt; The agent can act only while armed — a time-boxed state a human turns on from the tray. Disarmed means inert, no matter what any connected agent asks for. (In the current version, approval is one click to arm, and actions inside the armed window are pre-approved — which makes the Arm decision itself the safety-critical one, and I treat it that way.)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Read-back verification.&lt;/strong&gt; After acting, the tool reads the resulting state and reports what actually happened — done, already done, unknown, or failed — instead of assuming a click landed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A full log.&lt;/strong&gt; Every action is recorded, because "the agent did something while I wasn't looking" is only acceptable if I can replay exactly what.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;An earlier experiment is worth naming because it failed in an instructive way: synthetic pointer injection. Technically it worked. Practically, it moved the real cursor and stole the foreground — the exact bug I was trying to fix — so I threw it out. The most dangerous component in a design like this is the one that "works" by breaking your core guarantee.&lt;/p&gt;

&lt;h2&gt;
  
  
  About games and anti-cheat, explicitly
&lt;/h2&gt;

&lt;p&gt;To be clear about what this is &lt;em&gt;not&lt;/em&gt;: it is not a game automation tool, it never interacts with a game process, and my rule is that agent operation stays out of protected matches entirely — the tool is excluded from my gaming setup's resident startup, and I exit it before playing anything with strict anti-cheat.&lt;/p&gt;

&lt;p&gt;The anti-cheat relevance runs the other direction: the design is &lt;em&gt;boring on purpose&lt;/em&gt;. No driver, no virtual HID, no input injection, no process interference. Those absences are the feature.&lt;/p&gt;

&lt;h2&gt;
  
  
  When you should NOT build this
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Several humans, one PC?&lt;/strong&gt; Buy MouseMux.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You need a real security sandbox?&lt;/strong&gt; Use a VM. A background-input tool in your own session is isolation of &lt;em&gt;attention&lt;/em&gt;, not of &lt;em&gt;trust&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your apps are games, custom-drawn canvases, or anything that ignores window messages and exposes no accessibility tree?&lt;/strong&gt; Background input will mostly fail, and failing honestly is all you'll get.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A headless browser or an API can do the job?&lt;/strong&gt; Use that. Desktop automation is the last resort, not the first.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Build your own only if your constraints look like mine: one PC, one human who refuses to give up the mouse, agents that must act in real desktop apps, and a hard requirement that "no" is an acceptable answer from the tool.&lt;/p&gt;

&lt;h2&gt;
  
  
  What's next
&lt;/h2&gt;

&lt;p&gt;In a follow-up I'll go through the failure catalogue in detail — the approaches above, what each one actually did when measured, and the acceptance checks I use to prove the real pointer never moved.&lt;/p&gt;

&lt;p&gt;If you're evaluating MouseMux (or Anode, NexusWindows, agent-desktop, Cua Driver — this space got busy fast) for an agent use case, I'd genuinely like to hear where my decision tree diverges from yours.&lt;/p&gt;




&lt;p&gt;*Draft v1 — 2026-11-19. Personal project, built and run on my own &lt;/p&gt;

</description>
      <category>ai</category>
      <category>windows</category>
      <category>automation</category>
      <category>mcp</category>
    </item>
  </channel>
</rss>
