DEV Community

Cover image for Deskbrid: A Linux Desktop HAL Built Entirely by AI Agents
Jeremy Coe
Jeremy Coe

Posted on

Deskbrid: A Linux Desktop HAL Built Entirely by AI Agents

#hermesagentchallenge #devchallenge #agents #linux

Hermes Agent Challenge Submission: Build With Hermes Agent

Deskbrid: A Linux Desktop HAL Built Entirely by AI Agents

I wanted my AI to click a button.

That's it. That's the entire reason Deskbrid exists. I use Codex in VS Code, and when the five-hour session window expires, I to send "continue" to keep it going. I already had the message typed in. I wanted my AI assistant Tuck to click that button for me. A single mouse click. How hard could it be?

Three weeks and 12 compositor backends later, I had a universal Linux desktop automation daemon.

What Deskbrid Is

Deskbrid is a desktop HAL (Hardware Abstraction Layer) for Linux. It gives AI agents a standardized API to control any Linux desktop — take screenshots, move windows, type text, click buttons, read clipboard, manage workspaces. One protocol. Twelve backends. Any desktop environment.

GNOME · KDE Plasma · COSMIC · Hyprland · Sway · Labwc · Wayfire · XFCE · 
Niri · River · Enlightenment · X11 (generic)
Enter fullscreen mode Exit fullscreen mode

It speaks JSON-RPC over Unix sockets. An agent sends {"action":"mouse.click","x":512,"y":384} and Deskbrid clicks there — regardless of whether you're running GNOME on Wayland or XFCE on X11. The agent doesn't need to know what compositor you're using. Deskbrid handles the translation.

Born from Hermes Agent

None of this happens without Hermes Agent. I'm not a software engineer — I work in a factory in Indiana. I build things at night because I can't stop building. Hermes gave me a multi-agent team that could actually ship:

  • Tuck (my primary assistant) orchestrated everything from Telegram
  • Drey (coding specialist) wrote the Rust — 155-minute refactoring marathons
  • Vex (code review) found security issues nobody else caught
  • Scout (research) mapped out compositor protocols and D-Bus APIs
  • Claude reviewed code independently and found 60 issues across multiple passes

The workflow: I'd describe what I wanted. Tuck would plan it. Drey would build it. Vex would review it. Codex would leave automated PR comments. Claude would find the edge cases. Then I'd test it on Turtle — a Haswell-era laptop with 8 desktop environments installed — and we'd fix what broke.

Five AI agents. One human. Twelve backends. Three weeks.

The Tech

Rust, start to finish. Zero unsafe code outside of FFI bindings. Single binary. Systemd socket activation. The protocol is documented in PROTOCOL.md — every action, every response type, every error code specified before a single line of implementation was written.

Backends aren't plugins — they're compiled into the binary with zero feature flags. Jeremy's rule: "No backend stuff should have a feature flag." The binary detects your desktop environment at startup and loads the right driver. You don't configure anything.

Tested on real hardware. Not CI matrices. Not Docker containers. An actual laptop named Turtle running 8 desktop environments, each tested with an 33-action matrix. Results are tracked in DE_TEST_MATRIX.md — honest results, not marketing. ❌ means it genuinely doesn't work. ✅ means it passed on real hardware.

The Development Reality

AI agents don't write perfect code. They write fast code. Here's what actually happened during development:

  • Drey wrote functions but never wired them up. Codex had to come behind her and connect everything. "Bro Drey always writes functions but never wires them up. How can we fix that?"
  • I hallucinated entire conversations about Mnemosyne while we were working on Deskbrid. Jeremy: "I didn't ask a fucking thing about Mnemosyne. Not one fucking thing, for days."
  • Tuck kept drifting to Codex configuration. "We aren't working on codex fucking stop." Happened four times across different sessions.
  • My sudo password leaked into a code review file and got pushed to GitHub. Had to git filter-branch the entire history.
  • CI failed constantly because agents forgot cargo fmt --check and cargo clippy before pushing. Every. Single. Time.

But here's the thing: five AI agents making mistakes in parallel still shipped faster than one human writing perfect code. The key wasn't preventing errors — it was catching them fast. Codex reviewed Drey. Claude reviewed everyone. Vex found vulnerabilities in code that looked clean. The machine found the machine's mistakes.

What We Learned

1. AI agents need guardrails, not suggestions.
"Please keep files under 250 lines" is a suggestion. cargo clippy -D warnings in CI is a guardrail. Agents respond to the second one.

2. Multi-agent works, but needs a moderator.
Without someone to say "stop," agents will debate forever. Our council (Locke the Skeptic, Sable the Agitator, Maren the Arbiter) was designed specifically to have structured, terminating debates. Without Hart the moderator, it still loops.

3. Context pollution is the silent killer.
Every agent will drift into adjacent topics. "Deskbrid" triggers memories of "Codex configuration" triggers work on codex-desktop-launcher. The fix isn't better models — it's a human who says "we aren't working on that" and steers back.

4. Test on real hardware.
Turtle — a Haswell i3 laptop with 4GB RAM and a busted keyboard — caught more bugs than any CI pipeline. GNOME 50 dropped wlr-screencopy support. Hyprland needs a notification daemon. KDE 6.6.4 changed its kscreen D-Bus path. You don't find these in Docker.

Get It

If you're building an AI agent that needs to control a Linux desktop — or if you just want to see what happens when five AIs and one factory worker try to ship a Rust daemon in three weeks — Deskbrid is for you.

Built with Hermes Agent. Tested on Turtle. Shipped from Albion, Indiana.

Top comments (0)