We Kept Rebuilding Aiden for Six Months: From a HID Demo to a Physical AI Agent

Six months ago, Aiden was much closer to a hardware experiment than an Agent platform.

The first public version proved a simple idea:

1. Capture the screen of a phone.
2. Send that image to a multimodal model.
3. Turn the model’s decision into keyboard, pointer, or touch input.
4. Observe the result and repeat.

That was enough for a demo.

It was not enough for a system that had to survive USB reconnects, stale screenshots, context limits, interrupted voice conversations, different operating systems, and tasks that occasionally needed a human.

During roughly six months of development—the public repository history begins in late April—we kept rebuilding Aiden around those failures.

This post is a tour of the current architecture, but more importantly, it is a record of the assumptions we had to replace along the way.

What Aiden actually is

Aiden is a physical AI Agent project for operating phones and computers through an external hardware control path.

The current development-board implementation connects to a target device through a USB-C hub:

• the target display is captured through HDMI;
• an RK628D or TC358743 bridge converts the video signal to CSI;
• a C++ service captures frames from the Linux video device;
• the Go Agent sends visual observations to a configured multimodal model;
• the model returns structured tool calls;
• the Agent writes keyboard, pointer, touch, or auxiliary-control reports through Linux USB HID gadget devices.

The basic loop looks like this:

Target display
    |
    v
HDMI -> RK628D or TC358743 -> CSI
    |
    v
/dev/video0
    |
    v
frame_service
    |
    v
Screenshot observation
    |
    v
Go Agent -> configured model endpoint
    |
    v
Structured tool call
    |
    v
/dev/hidg0, /dev/hidg1, /dev/hidg2
    |
    v
Target device input

Voice adds another path:

Board microphone
    |
    v
audio_service
    |
    v
VAD -> STT or audio attachment
    |
    v
Go Agent -> LLM
    |
    v
TTS
    |
    v
audio_service playback

The firmware and device-side Agent Runtime are open source. The current repository includes the firmware overlay, C++ hardware services, Go Agent, OTA tooling, tests, and benchmark infrastructure.

It is not the final integrated Aiden hardware product.

There is another boundary worth making clear:

The Agent Runtime runs on the device. Model inference does not have to.

Screenshots, audio, and text are sent to the model, STT, TTS, and search endpoints configured by the owner. Those endpoints can be hosted or self-hosted. Aiden does not require an Aiden-hosted inference backend.

You can inspect the current architecture in the Aiden documentation and the open-source firmware repository.

The architecture we ended up with

The current system has several distinct layers:

Layer	Responsibility
Hardware abstraction	Low-level GPIO, audio, video, and HID capabilities
Hardware services	Own shared resources such as video capture and audio
Cross-language transport	Connect Go and C++ through Unix domain sockets
Go Agent	Model requests, tools, memory, tasks, voice, and Web APIs
USB gadget layer	Keyboard, pointer/touch, auxiliary control, and ECM networking
Firmware integration	Startup scripts, configuration, watchdogs, OTA, and image generation
Evaluation	Environment bridges, benchmarks, traces, screenshots, and reports

Four principles now shape most of the codebase:

1. A hardware resource should have one owner.
2. Go and C++ should communicate through explicit protocols rather than a shared ABI.
3. Agent reasoning should be decoupled from hardware implementation details.
4. Every important action should leave enough state and evidence to explain what happened.

We did not begin with all four principles. We reached them by repeatedly breaking the earlier architecture.

Refactor 1: From opening hardware everywhere to one resource owner

Screen capture was one of the first architectural problems.

A simple prototype can open /dev/video0, capture a frame, and close it. That becomes unreliable when several tools or services believe they can access the same device.

Multiple consumers introduce questions that the model should never have to answer:

• Which process owns the capture stream?
• Who configures the sensor?
• What happens if HDMI disconnects?
• Which frame is fresh?
• Who restarts the pipeline after failure?

We moved that responsibility into frame_service.

frame_service is the single owner of the video device. Other processes request screenshots through a Unix domain socket rather than opening the hardware themselves.

The audio path follows the same principle. audio_service owns recording, playback, and volume state.

The UDS protocol uses a JSON header with an optional binary payload. That gives us a language-neutral boundary: C++ can own the hardware while Go consumes the capability without depending on a C++ ABI.

Go Agent
   |
   | screenshot request over UDS
   v
frame_service
   |
   | fresh frame
   v
/dev/video0

This separation also made failures easier to locate.

If the Agent cannot get a screenshot, we can ask whether the failure is in:

• the model;
• the Go screenshot client;
• the UDS protocol;
• frame_service;
• the HDMI bridge;
• or the underlying video device.

Before the boundary existed, all of those failures looked like “the screenshot tool failed.”

The first major frame-service work appeared in PR #17. Later changes moved screenshots behind a Screen Provider, changed capture to happen on demand, and improved freshness and latency in PR #519, PR #521, and PR #559.

The reusable lesson was:

Don’t let every part of an Agent application become a hardware driver.

Refactor 2: From platform branches to explicit action semantics

Our early HID code was full of apparently small platform decisions.

Should a pointer be absolute or relative? Is a swipe a mouse movement or a touchscreen gesture? Which coordinate space does a screenshot use? Should text entry use HID, ADB, or a Phone Bridge?

These decisions leaked into tool descriptions, prompts, configuration, and benchmark code.

That made the Agent’s behavior depend on several sources of truth at once.

We eventually separated two layers:

• what the Agent intends to do;
• how the configured device performs it.

The intention layer contains operations such as:

• click;
• swipe;
• drag;
• keyboard shortcut;
• text entry;
• auxiliary media or Android controls.

The execution layer selects a configured HID, ADB, or HTTP input path and applies the appropriate coordinate and keyboard rules.

This became the MNK Provider abstraction.

We also removed reverse platform inference. The configured device_type became the authority instead of asking multiple parts of the runtime to guess the target platform.

That made the system less magical and more predictable.

Aiden still does not have one universal input transport. HID, ADB, and HTTP remain different paths with different constraints. The abstraction unifies the Agent-facing action semantics, not the underlying operating systems.

We then changed what happens after an action.

A click or swipe should not be considered successful merely because writing the input report returned no error. The device may ignore the event, display another page, or route the input somewhere unexpected.

The runtime began attaching post-action visual observations and touch markers so the model could reason about where the action landed and what changed afterward.

A drag could also be separated into:

drag_start
    |
    v
observe the screen
    |
    v
drag_release

That lets the Agent inspect the intermediate state before committing to the release.

The central input refactors are visible in PR #539, PR #543, PR #588, and PR #595.

The lesson:

A shared action interface is useful only if the platform-specific boundary underneath it remains explicit.

Refactor 3: From scattered settings to a configuration control plane

As Aiden added more model, STT, and TTS providers, configuration became a source of architectural debt.

Each integration brought slightly different assumptions:

• provider-specific base URLs;
• different model identifiers;
• different reasoning controls;
• credentials that must survive unrelated edits;
• STT and TTS settings with their own defaults;
• fields understood by the runtime but not yet understood by the UI.

A naive configuration page reads a file, edits the fields it knows, and writes the whole file back.

That is dangerous. Unknown fields, comments, ordering, and credentials can disappear because the UI did not understand them.

We gradually moved provider and configuration ownership into the Agent Runtime.

The current direction includes:

• unified model, STT, and TTS provider records;
• provider-owned endpoint configuration;
• metadata-driven validation;
• native Anthropic and Responses API paths;
• provider-aware reasoning controls;
• JSON Merge Patch updates;
• atomic TOML writes;
• preservation of comments, formatting, unknown fields, and write-only credentials.

Config Web also moved toward using the Go Agent as its management API instead of maintaining a second implementation of configuration behavior.

That is more than a frontend refactor.

It establishes one authority for resolving, validating, and applying runtime configuration.

Relevant changes include PR #484, PR #504, PR #542, PR #551, and PR #631.

The lesson:

A configuration UI is part of your runtime contract, not just a form that edits a file.

Refactor 4: From one growing transcript to several kinds of memory

Our early mental model of context was straightforward: keep the conversation and send the relevant history back to the model.

That stops being straightforward when the history contains:

• user messages;
• assistant responses;
• screenshots;
• tool calls;
• large tool results;
• device state;
• runtime warnings;
• learned procedures;
• user preferences;
• task evidence.

Putting all of this into one transcript creates both token pressure and semantic confusion.

We started separating the information by purpose.

The current memory architecture distinguishes:

• session memory, which keeps the active conversation and compressed history;
• long-term memory, which stores explicitly saved user facts, preferences, and rules;
• device memory, which stores reusable device, application, navigation, procedure, calibration, failure, and fact knowledge;
• task episodes, which preserve execution evidence for audit and background learning.

Device memory is no longer automatically injected into every prompt. The model calls the recall tool when a task materially depends on previous device knowledge.

The runtime message model also distinguishes ordinary conversation from state generated by the system:

const (
    MessageRoleUser       MessageRole = "user"
    MessageRoleAssistant  MessageRole = "assistant"
    MessageRoleToolCall   MessageRole = "tool_call"
    MessageRoleToolResult MessageRole = "tool_result"
    MessageRoleState      MessageRole = "state"
    MessageRoleSystem     MessageRole = "system"
    MessageRoleNotice     MessageRole = "notice"
)

A State message can describe the current device or screen environment.

A Notice message can carry a task result, loop-guard correction, or human-action request without pretending that the person wrote it.

Large results and old context introduced another set of refactors:

• large tool results can be stored as recoverable artifacts;
• historical tool results and state can be pruned before compaction;
• a context session can rotate after truncation;
• parent-child lineage records how the new session was created;
• persisted session events became the recovery source instead of redundant history mirrors.

Relevant changes include PR #475, PR #497, PR #524, PR #607, PR #609, and PR #618.

The current design is documented in the Aiden memory plane.

The lesson:

Context management is part of Agent architecture, not cleanup performed after the architecture is finished.

Refactor 5: From voice input to foreground and background Agents

Adding speech to an Agent looks easy when voice is treated as another input method:

speech -> STT -> Agent -> TTS

That works until the Agent starts a multi-step device task.

A device task may need to capture several screens, call tools, wait for UI changes, and recover from unexpected states. Meanwhile, the person may want to interrupt:

• “Stop.”
• “Use the other account.”
• “What is happening?”
• “I’ll enter the password myself.”

If conversation and execution share one loop, the conversation waits for the task. If every backend event is immediately sent to the voice model, the Agent can interrupt itself.

We rebuilt realtime voice around two cooperating Agents:

• a foreground Realtime Agent that owns the live conversation;
• a backend Agent that executes device tasks;
• an asynchronous task queue connecting them.

The task manager uses an explicit lifecycle:

type Status string

const (
    StatusCreated    Status = "created"
    StatusQueued     Status = "queued"
    StatusRunning    Status = "running"
    StatusCancelling Status = "cancelling"
    StatusCancelled  Status = "cancelled"
    StatusFailed     Status = "failed"
    StatusCompleted  Status = "completed"
)

The status describes execution state. It does not prove that the user’s intended goal was achieved.

The foreground Agent can create, query, or cancel background tasks. It does not need to wait for the backend Agent to finish before continuing the conversation.

Human handoff also became part of the task lifecycle.

If a password, CAPTCHA, payment confirmation, permission dialog, or ambiguous decision requires a person, the backend task can publish a pending action and remain in the running state.

After the person finishes, the foreground Agent returns a continuation message. The manager resumes the same task identity and backend context instead of starting an unrelated task.

We also added backpressure to result delivery. Terminal task updates use a 500-millisecond sliding debounce window, and the runtime injects them only when the foreground is idle.

This prevents several nearby results from creating several competing spoken responses.

The first full-duplex implementation landed in PR #566. Realtime provider adapters and the expanded foreground tool surface followed in PR #620 and PR #623.

The architecture is described in Foreground and Background Agents.

The lesson:

A voice Agent operating a physical device needs ownership and scheduling rules, not only lower audio latency.

Refactor 6: From successful demos to saved execution evidence

A demo video tells you that one task worked once.

It usually does not tell you:

• which tools were called;
• whether the Agent used the intended input path;
• whether the screenshot was fresh;
• what the screen looked like before and after;
• whether the final response matched the physical outcome;
• whether the result can be reproduced.

We built the benchmark system around an Environment Bridge contract:

Runner
   |
   v
Go Agent
   |
   v
Environment Bridge
   |
   +-- physical device
   +-- MobileGym / ADB
   +-- desktop environment
   +-- custom bridge

For each task, the runner can:

1. prepare an isolated environment;
2. capture a pre-task screenshot;
3. send the task to the real Agent;
4. capture a post-task screenshot;
5. extract the structured tool trace;
6. run deterministic assertions;
7. optionally evaluate the saved evidence with a judge;
8. retain the artifacts for later analysis.

Execution and scoring are separate. A rubric or judge can change without forcing the physical task to run again.

We also added:

• a hardware-free Docker sandbox;
• centralized benchmark configuration;
• local low-level smoke tests;
• macOS, Linux, and Windows desktop bridges;
• environment health checks;
• isolated Agent workers for concurrent environments;
• separate pre/post screenshots and structured traces.

The benchmark now asks a better question than “Did the demo look good?”

It asks:

What did the Agent observe, what action did it take, what changed, and what evidence supports the reported result?

Relevant changes include PR #523, PR #533, PR #569, PR #593, and PR #612.

The current flow is documented in the benchmark architecture.

Some refactors made Aiden smaller

Not every important change added a feature.

We removed several ideas after they stopped matching the system we were building:

• reverse platform inference;
• automatic upfront memory injection;
• unused chat-history mirrors;
• duplicated configuration paths;
• obsolete script and image-diff tools;
• tool descriptions containing platform policy that belonged in configuration;
• assumptions that a completed loop implied a successful physical outcome.

Removing a tool can look like lost capability.

For an Agent, it can be the opposite. Every exposed tool expands the model’s decision space. An ambiguous or obsolete tool makes the runtime harder to understand and the Agent harder to control.

One example is PR #601, which removed script and image-diff tools while preserving post-action screen-change detection as a narrower capability.

The lesson:

A smaller, explicit tool surface can be more capable than a larger, ambiguous one.

What the six months changed

The visible result is a larger system, but the more important change is in where responsibilities now live.

Earlier assumption	Current direction
Any process can capture a frame	One service owns each hardware resource
Tools decide platform behavior	Configuration selects an explicit provider
A written HID report means success	Observe and verify the resulting screen
Configuration is a file-editing problem	The Agent owns the runtime contract
Context is one transcript	Conversation, state, memory, and evidence are separate
Voice is another input mode	Foreground conversation and backend work are decoupled
A stopped loop means success	Execution state and physical outcome are different
A demo is enough evidence	Save screenshots, traces, assertions, and reports
More tools mean more capability	Smaller boundaries can improve reliability

None of these principles is unique by itself.

The difficulty is applying all of them at once while a model is interacting with a real screen, an operating system, a USB stack, and hardware that can disconnect.

A physical Agent is not only an LLM with tools.

It is a distributed system squeezed onto a small device.

Where the project stands

The current Aiden development board can combine:

• HDMI screen observation;
• keyboard, pointer, touch, and auxiliary USB HID control;
• USB ECM networking;
• configurable model, STT, and TTS endpoints;
• voice interaction;
• skills and several kinds of memory;
• foreground and background task execution;
• human handoff;
• OTA and diagnostics;
• device and desktop benchmark environments.

There are still real boundaries.

The basic control path requires a target capable of video output and USB HID input. iOS requires AssistiveTouch. Optional Phone Bridge, notification, BLE, and ADB paths have their own setup requirements.

Compatibility still depends on the target device, operating-system version, capture bridge, USB behavior, audio path, permissions, and configured inference endpoints.

The current repository should be read as a development-board implementation and an invitation to inspect the engineering—not as a claim that every supported-looking combination has already been validated.

If you are building voice Agents, GUI Agents, embedded controllers, or hardware automation systems, we would especially value reports that include:

• the board and firmware revision;
• the target device and OS version;
• the task prompt;
• the configured model endpoint;
• screenshots before and after;
• the structured tool trace;
• interruption or handoff behavior;
• what you expected and what actually happened.

You can explore the project through:

• Aiden Firmware on GitHub
• System Architecture Overview
• Hardware and Wiring Guide
• Agent Documentation
• Benchmark Documentation

We are still refactoring Aiden.

That is not because the original idea stopped working.

It is because “the Agent moved the cursor” and “the system can explain what it observed, what it did, what happened next, and why it stopped” are two very different engineering milestones.